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Appendix  5 

Analysis  of  Data  from  the  OIG  Case  Information 
Management  System  (CIMS) 


5.0  Introduction 

Many  of  the  direct  monetary  savings  associated  with  Ojjeration  Restore  Trust  (ORT)  ultimately  result  from 
investigations  by  the  Office  of  Investigations  (01)  of  suspect  providers.  01  conducts  cnminal.  civil,  and 
administrative  investigations  of  providers  suspected  of  fraud  and  abuse.  Their  investigations  result  in 
criminal  convictions,  civil  judgements,  monetary  recovenes.  and  exclusions  of  providers  from  the 
Medicare  program. 

These  investigations  have  several  sources  of  potential  savings  for  the  Medicare  program  that  are  recorded 
in  OI's  Case  Information  Management  System  (CIMS): 

•  Direct  monetary  recoveries  from  providers  (i.e.,  from  fines,  settlements,  judgments, 
recoveries,  or  restitution) 

•  The  avoidance  of  costs  (e.g..  by  terminating  the  improper  obligation  of  Medicare  funds). 
These  expected  program  savings  represent  future  costs  that  Medicare  was  able  to  avoid  due 
to  OI's  actions  against  the  provider. 

ORT  had  a  large  impact  on  the  volume  and  sources  of  cases  opened  by  01: 

The  number  of  non-exclusion  cases  increased  from  1 64  to  265  during  ORT,  a  62%  increase. 
Given  the  decrease  in  volume  observed  for  comparison  states,  ORT  was  associated  with  a 
103%  net  increase  in  volume. 

Volume  increased  in  all  five  demonstration  states.  New  York  was  responsible  for  more  than 
half  of  the  increase,  with  volume  of  non-exclusion  cases  increasing  from  53  in  the  pre-ORT 
period  to  1 1 1  dunng  ORT. 

The    proportion    of    cases    referred    by    the    Hotline,    state    agencies,  and 
beneficianes/employees/citizens  increased  dunng  ORT.  There  was  also  an  increase  in 
the  number  (though  not  the  proportion)  of  cases  referred  by  HHS,  HCFA,  and  HCFA 
contractors. 

A  major  evaluation  question  is  how  much  savings  result  from  the  extra  resources  given  to  Office  of  the 
Inspector  General  (OIG)  offices  under  ORT.  In  asking  this  question,  we  need  to  emphasize  that  OI 
investigations  are  not  a  product  of  01  efforts  alone.  Some  investigations  may  be  the  result  of 
collaborative  efforts  with  the  Office  of  Audit  Services  (OAS),  the  Office  of  Evaluations  and  Inspections 
(OEI),  the  Health  Care  Financing  Administration  (HCFA),  the  Administration  on  Aging  (AOA),  and/or 
other  ORT  and/or  other  non-ORT  agencies  such  as  Medicaid  Fraud  Control  Units  (MFCUs),  the  Federal 
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Bureau  of  Investigation  (FBI),  or  the  Department  of  Justice  (DOJ).  We  focus  major  evaluation  efforts  on 
01  in  part  because  the  efforts  of  others  lead  to  OI  case  development,  and  it  is  at  the  point  of  case 
development  that  the  savings  are  consummated.  In  addition,  most  ORT  resources  have  gone  to  01.  It 
is  therefore  essential  to  ask  how  much  savings  have  resulted  from  these  additional  resources. 

Estimation  of  savings  posed  a  number  of  significant  evaluation  challenges.  Many  cases  are  still  pending, 
and  it  is  not  possible  to  know  what  the  ultimate  outcome  of  these  cases  will  be.  It  is  not  obvious  how 
cases  that  opened  before  ORT  but  that  continued  development  dunng  the  demonstration  should  be  treated. 
The  origination  of  these  cases  was  not  affected  by  ORT.  but  ORT  may  have  influenced  the  ultimate 
outcome  of  the  case.  In  addition,  data  on  actual  Medicare  recovenes  (as  opposed  to  accounts  receivables 
established  as  a  result  of  01  cases)  are  incomplete.  Because  of  these  difficulties,  we  present  a  range  of 
estimates  of  the  impact  of  ORT  on  receivables  and  actual  Medicare  recovenes. 

Our  range  of  estimates  of  ORT  savings  ranges  from  SO. 5  million  (the  estimated  Medicare  recovenes  for 
cases  opened  dunng  ORT  (our  most  narrow  definitions  of  both  ORT  and  ORT  savings)  to  S 1 1 7.7  million 
(using  a  broader  definition  of  what  constitutes  an  ORT  case  and  estimating  the  expected  receivables  from 
pending  cases).  As  of  May  1997.  ORT  was  associated  with  an  additional  $50  million  in  identified 
receivables  from  cases  that  opened  during  the  demonstration  penod. 

In  this  appendix,  we  describe  the  data  sources  and  evaluation  methods  used  to  estimate  the  impacts  of 
ORT,  and  then  present  estimates  of  the  impacts  associated  with  ORT.  Our  estimates  of  ORT  effects  are 
based  on  comparisons  of  1 )  ORT-subject  type  cases  opened  in  ORT  states  to  2)  cases  from  the  pre- 
demonstration  period  and  from  a  group  of  comparison  states. 

5.1    Data  Source  and  Evaluation  Design 

In  this  section,  we  describe  01' s  case  tracking  system,  which  is  the  data  source  we  used  in  estimating  the 
impacts  of  ORT.  We  also  describe  our  approach  for  estimating  the  impacts  of  ORT. 

5.1.1     Data  Source:  Ol's  Case  Information  Management  System  (CIMS) 

The  outcomes  of  01  investigations  are  tracked  by  the  CIMS.  an  automated  status  and  tracking  system  used 
to  record  01  case  development  and  outcomes.  CIMS  tracks  cases  from  the  time  a  complaint  is  first 
received  until  the  date  a  final  outcome  for  the  case  occurs.  CIMS  contains  detailed  infonmation  on  case 
sources,  subject  type,  and  location,  as  well  as  the  dates  of  major  case  development  milestones  (e.g..  when 
the  case  opened,  when  it  was  referred  to  prosecutors,  when  it  went  to  tnal),  and  the  final  outcomes  of  cases 
(e.g.,  exclusions,  prison  sentences,  and  the  establishment  of  receivables). 

We  used  data  from  two  components  of  the  CIMS  system.  The  Case  File  contains  information  from  the 
CIMS  Situation  Report,  which  is  used  to  handle  complaints  received  (i.e..  before  an  official  case  is 
opened)  and  is  completed  before  any  investigative  work  is  done.  The  data  in  the  Situation  Report  include 
basic  descriptive  information  such  as  the  01  suboffice  handling  the  complaint,  the  source  and  sub-source 
for  the  case',  the  allegation,  the  subject  type,  and  (for  complaints  that  do  not  result  in  actual  cases),  the 


1      For  cases  referred  by  a  Federal  government  agency,  the  sub-source  identifies  the  specific  agency  that  referred  the  cotnplaint. 
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date  and  reason  that  the  complaint  was  disposed  of  administratively.  The  Case  File  also  contains  case- 
level  development  information  from  the  CIMS  Action  Summaries  (referral  information,  joint  agency 
involvement,  date  the  case  opened,  and  the  date  the  case  closed). 

The  CIMS  Subject  File  tracks  case  development  milestones  and  outcomes  for  each  subject  involved  m  a 
given  case.  Since  some  cases  have  multiple  subjects,  the  Subject  File  can  contain  multiple  records  for  a 
single  case.  For  cases  with  multiple  subjects,  we  linked  together  records  for  all  subjects  m  the  case  to 
create  a  case-level  file  that  contains  all  of  the  development  milestones  and  outcomes  for  the  case.  The 
Subject  File  contains  fields  from  the  Criminal,  Civil.  Sanction,  Administrative,  and  Civil  Monetarv  Penalty 
Law  of  1981  (CMPL)  Action  Summaries.  The  Action  Summanes  are  used  to  record  all  action  taken  on 
cases. 

The  CIMS  data  used  for  this  report  contain  information  on  2.271  ORT-related  cases  "(excluding 
complaints  that  were  disposed  of  administratively  and  did  not  result  in  the  opening  of  a  case)  that  opened 
between  January  1991  (a  somewhat  arbitrary  start  date  that  was  early  enough  to  include  a  substantial  pre- 
ORT  penod)  and  March  31,  1997.  the  last  date  of  ORT.  Of  these,  952  were  exclusion  cases. 

5.1.2  Methods 

Our  basic  approach  for  estimating  the  impacts  of  ORT  on  the  volume  and  outcomes  of  01  cases  is  to 
compare  measures  of  change  (from  the  pre-ORT  to  the  ORT  penod)  for  ORT  states  and  a  group  of 
companson  states'.  This  type  of  model  adjusts  for  secular  trends  that  may  have  affected  pre/post  changes 
in  ORT  states  even  in  the  absence  of  the  demonstration.  The  assumption  underlying  this  approach  is  that 
any  change  that  might  have  occurred  around  the  time  ORT  staned  (e.g..,  changes  due  to  the  split  between 
Social  Secunty  Administration  (SSA)  and  the  Department  of  Health  and  Human  Services  (DHHS)  or  the 
streamlining  implemented  in  1994  and  1995  by  the  Inspector  General  that  resulted  in  the  early-out  and 
buyout  of  management  positions  and  the  closing  of  17  OI  suboffices')  had  the  same  effects  in  both 
demonstration  and  comparison  states.  This  approach  differs  from  that  used  for  the  ORT  Performance 
Indicators,  which  are  restricted  to  the  ORT  period  and  ORT  states,  and  do  not  use  either  pre/post  or 
demonstration/comparison  state  techniques  to  measure  the  impact  of  ORT. 

ORT  has  certain  characteristics  that  complicate  our  evaluation  of  its  impacts.  In  particular,  since  cnminal 
investigations,  especially  those  that  result  in  referral  to  a  prosecutor,  often  take  several  years  to  reach  a 
final  outcome,  many  of  the  savings  from  ORT  projects  have  not  yet  been  obtained,  even  if  they  are  likely. 
In  addition,  many  cases  that  opened  before  ORT  started  have  continued  development  under  ORT,  and  it 
is  not  clear  whether  such  cases  should  be  counted  as  ORT  or  pre-ORT,  as  these  cases  have  some 
characteristics  (and  doubtless  bear  some  of  the  infiuence)  of  both  periods.  In  the  sections  below,  we 
review  some  of  the  key  choices  we  have  made  in  structunng  our  analysis. 


2  -ORT  relaled  -cases  include  those  wuh  an  ORT  subject  type  (DME  supplier,  home  health  agency,  hosptce.  nursing  home)  and  cases  that 
were  identified  as  ORT-related  in  CIMS. 

3  Note  that  in  our  analysis  of  cases  identified  as  ORT-relaled  in  CIMS.  we  did  not  use  a  double  difference  model  to  estimate  ORT  effects, 
because  it  was  not  possible  to  define  either  a  companson  group  or  a  pre-ORT  penod  for  these  cases. 

4  See  the  Office  of  the  Inspector  General's  Semiannual  Report  (October  1994  -  March  199M  for  details. 
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Defining  Comparison  Areas 


Selection  of  comparison  states  is  complicated  by  tiie  non-randomized  nature  of  ORT  and  the  spillover  of 
ORT  effects  into  non-ORT  states.  The  ORT  states  were  deliberately  selected  for  the  likely  impact  of  an 
ORT-style  effort.  Other  slates  differ  from  the  ORT  states  m  important  ways  besides  the  presence  of  the 
intervention,  making  comparisons  with  other  states  problematic. 

For  most  of  the  analyses  described  in  this  section,  we  compared  outcomes  observed  in  ORT  states  to  those 
observed  in  a  group  of  comparison  states.  Companson  states  included  all  non-ORT  states  with  an  01 
suhoffice  (Exhibit  5-1 ).  We  are  thus  taking  the  presence  of  a  sub-office  as  a  proxy  for  features  we  wish 
to  hold  common  between  ORT  and  comparison  areas,  such  as  having  a  large,  predominantly  urban 
population.  This  rule  excludes  such  states  as  Montana.  Vermont,  Idaho.  Alaska,  and  Nebraska.  The  rule 
was  a  simple  way  to  avoid  having  our  subjective  judgement  determine  the  inclusion  or  exclusion  of 
particular  states.  Note  that  we  did  make  one  subjective  judgment.  Because  of  its  status  as  a  "quasi-ORT" 
state,  we  excluded  all  cases  from  Louisiana  from  our  analyses,  since  it  was  not  clear  whether  these  cases 
should  be  counted  as  ORT  or  non-ORT,  in  view  of  the  efforts  of  the  Texas  Regional  Office  in  Louisiana. 

Cases  were  assigned  to  states  based  on  the  state  field  in  the  CIMS  Situation  Report.  Some  cases  had 
subjects  from  more  than  one  state.  For  these  cases,  the  state  was  assigned  based  on  the  location  of  the  01 
suboffice  responsible  for  the  investigation. 

One  subsidiary  issue  in  our  analysis  is  that  resources  within  a  region  may  have  been  diverted  from  non- 
ORT  states  in  order  to  focus  on  ORT  projects.  For  example,  if  the  volume  of  cases  opened  in  Texas 
increased,  but  decreased  by  a  similar  magnitude  in  other  states  in  Region  VL  it  seems  reasonable  to  infer 
that  much  of  the  impact  observed  in  the  ORT  states  was  actually  the  result  of  diversion  of  efforts  from 
other  states  in  the  region.  For  that  reason,  we  performed  a  sub-analysis  comparing  outcomes  observed  in 
ORT  states  to  those  in  non-ORT  states  within  ORT  regions.^ 

ORT  effects  may  not  have  been  confined  to  demonstration  states.  Some  ORT  projects  explicitly  included 
non-ORT  states.  As  noted  above,  the  HFCA  office  in  Dallas,  for  example,  included  Louisiana  providers 
in  its  surveys  of  home  health  agencies.  01  in  Chicago  led  a  national  study  of  hospices.  OAS  projects  in 
New  York  included  three  projects  related  to  hospices  in  Puerto  Rico.  Each  of  these  projects  may  have 
resulted  in  referrals  to  01  and  additional  criminal  cases.  Assuming  that  most  of  the  leakage  of  ORT  effects 
into  non-ORT  states  is  confined  to  a  particular  region,  comparison  of  outcomes  observed  in  non-ORT 
states  within  ORT  regions  compared  to  non-ORT  regions  allows  us  to  take  account  of  ORT  effects  that 
are  realized  in  the  non-ORT  states  of  a  region.  If  substitution  of  resources  is  confined  within  the  region, 
then  companson  of  non-ORT  states  within  ORT  regions  to  non-ORT  regions  allows  for  a  measure  of  the 
impact  of  ORT  that  is  not  distorted  by  either  diversion  of  resources  or  leakage  of  ORT  effects  into 
neighboring  states. 


5      We  define  region  based  on  ihe  states  served  by  the  HCFA  Regional  Office. 
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Exhibit  5-1: 

Comparison  States  Used  in  ORT  Impact  Analyses 
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Defining  Cases 

Not  all  of  the  records  included  in  CIMS  were  used  in  these  analyses.  In  this  section,  we  descnbe  the 
cniena  used  for  selecting  which  case  records  to  include  in  our  impact  analyses. 

Except  for  subanalyses  where  we  use  OI's  definition  of  ORT  cases  (which  includes  138  cases  with  non- 
ORT  subject  types),  our  analyses  include  only  cases  for  which  one  or  more  subjects  were  identified  in  the 
CIMS  Situation  Report  as  being  an  ORT  subject-type  provider  (DME  supplier,  home  health  agency, 
hospice,  or  nursing  home).  Complaints  that  were  disposed  of  administratively  and  did  not  result  in  the 
actual  opening  of  a  case  were  excluded  from  our  analyses. 

"Exclusion  cases"  are  reported  in  CIMS.  but  they  are  fundamentally  different  from  cases  that  involve 
active  investigative  work  by  01.  An  "exclusion"  is  an  administrative  action  by  which  a  provider/supplier 
is  barred  from  receiving  Medicare  reimbursement  for  a  defined  penod  of  time.  01  processes  exclusion 
cases,  but  the  work  is  largely  administrative  and  does  not  typically  involve  active  investigative  work  by 
01  Special  Agents,  as  this  work  is  normally  a  routine  follow-up  to  findings  of  guilt  brought  about  by  the 
investigative  work  of  other  agencies,  such  as  MFCUs.  Because  exclusion  cases  tend  to  be  initiated  and 
investigated  by  other  agencies,  ORT  was  expected  to  have  a  different  impact  on  the  volume  of  exclusions 
(ORT  may  affect  the  number  of  exclusion  cases  since  the  agencies-e.g..  MFCUs,  HCFA  contractors-  that 
investigate  exclusion  cases  are  involved  in  ORT)  than  for  non-exclusion  cases,  which  involve  active 
investigations  by  01  of  alleged  criminal  activities,  and  often  result  in  successful  referral  to  prosecutors  and 
the  establishment  of  receivables. 


Defining  ORT-period  Cases 

There  is  no  one  single  definition  that  can  be  used  to  define  unambiguously  what  constitutes  an  ORT-period 
case.  In  this  report,  we  use  several  definitions  of  ORT  (Exhibit  5-2).  ORT  cases  are  defined  based  on 
the  subject  types  and  the  date  that  the  case  opened  (or  reached  a  final  outcome). 
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Exhibit  5-2     ORT  Definitions  for  CIMS  Analyses 


Name 


Description 


Uses 


0RT1 


Cases  from  ORT  states  and  subject  types   Use  for  all  outcomes  that  are  observed 


0RT2 


0RT3 


that  opened  dunng  the  demonstration 
period. 


All  cases  in  0RT1  plus  cases  that  opened 
in  the  comparison  penod  that  reached  a 
final  outcome  dunng  ORT. 


All  cases  identified  as  ORT-related  in 
CIMS  (i.e..  cases  where  Project 
identifier='PK',  regardless  of  state,  subject 
type,  or  time  penod.)  


when  the  case  opens  (volume,  subject 
types,  states,  sources)  and  to  estimate 
"pure-ORT"  estimates  of  the  impact  of 
ORT  on  receivables  identified  and 
Medicare  recovenes  (typically  "lower 
bound"  estimates  of  the  range  of  ORT 
savings). 

Use  in  conjunction  with  0RT1  for 
measunng  the  impact  of  ORT  on  case 
development  and  outcomes,  using 
"anificially  censored"  outcome  measures 
for  pre/post  comparisons  (see  discussion 
below) 

Use  to  report  GIMS-based  ORT 
performance  indicators 


Note:  ORTl  and  0RT2  only  count  as  ORT  related  those  cases  with  an  ORT  subiect  type  from  an  ORT  state. 


Defining  ORT  Cases  Based  on  Case  Development  Dates 

ORTl,  the  most  narrow  case  definition  we  used,  includes  only  those  cases  from  ORT  states*  and  subject 
types'  that  opened  during  the  demonstration  penod.  Cases  defined  as  ORT-related  using  this  definition 
can  be  regarded  as  "pure  ORT,"  since  both  the  ongination  and  the  development  of  the  case  occurred 
dunng  the  demonstration  period. 

This  definition  is  appropriate  for  examining  the  impact  of  ORT  on  measures  that  are  observed  when  a  case 
is  opened,  such  as  the  volume,  location,  subject  type,  and  sources  of  the  cases.  However,  the  majonty 
of  these  cases  (nationwide,  245  of  the  404  non-exclusion  cases  defined  as  ORT-related  using  the  ORTl 
definition)  were  still  pending  as  of  the  May  27,  1997,  data  cutoff  point.  As  a  result,  the  ORTl  definition 
is  less  useful  for  measuring  the  impact  of  ORT  on  the  identification  of  receivables  or  actual  Medicare 
recoveries.  Cases  that  opened  during  the  demonstration  penod  were  observed  for  a  maximum  of  26 
months  (i.e..  through  the  May  1997  cutoff  point  for  our  CIMS  extract),  and  many  were  obserxed  for  a 
much  shorter  period  of  time. 


6  Cases  were  assigned  to  a  stale  based  on  the  subject  s  stale  field  from  the  CIMS  Situation  Report.  For  the  few  cases  that  had  subjects 
from  multiple  stales,  the  state  was  assigned  to  the  case  based  on  the  OI  suboffice  responsible  for  the  investigation. 

7  For  cases  that  have  more  than  one  subject  type  (e.g..  not  ail  of  the  subjects  in  the  case  have  the  same  subject  type),  the  entire  case  was 
assumed  to  be  an  ORT  subject  type  if  al  least  one  subject  had  an  ORT  subject  type.  Hospice  cases  were  included  as  an  ORT  subject 
type. 
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A  broader  definition,  0RT2,  also  includes  cases  from  ORT  states  and  subject  types  that  were  open  but 
still  pendins>  when  ORT  started."  This  broader  definition  can  be  defended  to  the  extent  that  cases  that 
opened  prior  to  the  ORT  start  date  are  in  due  course  affected  by  ORT  (for  example,  through  the  additional 
agents,  resources,  and  travel  funds  provided  by  ORT).  The  ORT2  definition  is  useful  for  estimating  the 
impacts  of  ORT  on  case  development  and  case  outcomes.  Some  pending  cases  would  presumably  have 
had  a  successful  outcome  even  without  ORT.  We  descnbe  below  how  the  techniques  we  use  to  assign 
outcomes  to  the  ORT  and  comparison  penods,  allowing  us  to  estimate  the  marginal  effects  of  ORT  on 
cases  that  were  pending  when  the  demonstration  staned. 

The  third  definition  of  ORT  that  we  consider  includes  all  ca.ses  that  CIMS  explicitly  identifies  as  ORT- 
related.  These  cases  are  identified  by  the  "PK"  modifier  in  the  CIMS  project  identification  field.  Some 
of  these  cases  have  non-ORT  subject  types  or  states,  and  many  opened  before  ORT  staned.  It  is  not 
possible  to  construct  an  appropnate  comparison  group  for  this  subset  of  cases,  since  the  rules  for  inclusion 
and  exclusion  are  not  entirely  clear  from  the  CIMS  date  fields  alone.  The  subset  of  cases  identified  as 
ORT-related  in  CIMS  is  used  for  some  of  the  ORT  Pertormance  Indicators  developed  by  HCFA,  OIG.  and 
AOA.  As  a  result,  our  analysis  of  0RT3  cases  repons  Abt's  estimates  of  the  CIMS-based  performance 
indicators  resulting  from  these  cases. 

Defining  the  "Pre-ORT"  Period  for  Our  Analyses 

CIMS  includes  records  on  cases  that  opened  as  early  as  1977.  Older  cases  are  likely  to  differ  in  important 
ways  from  cases  that  opened  more  recently,  so  we  decided  to  exclude  older  cases  from  our  specification 
of  the  pre-ORT  penod.  The  pre-ORT  period  includes  cases  that  opened  in  the  24  month  period 
immediately  preceding  the  start  of  ORT  (Apnl  1 ,  1 993  -  March  31,1 995).'  These  cases  are  more  likely 
to  be  similar  to  ORT  cases  in  ways  other  than  the  presence  of  the  ORT  intervention,  and  reflect  trends 
towards  a  greater  awareness  of  fraud  and  abuse  issues  (e.g.,  through  the  South  Florida  Task  Force  or  the 
establishment  of  the  DMERCs)  that  were  in  place  prior  to  the  start  of  ORT. 

Evaluation  of  Censored  Case  Outcomes 

Criminal  and  civil  cases  often  take  three  years  or  more  to  reach  a  final  (or  even  a  probable)  outcome. 
Note,  for  example,  that  across  demonstration  and  companson  states,  only  39  ORT-period  cases  had  gone 
to  trial  and  achieved  a  monetary  outcome  as  of  May  28,  1997.  The  outcomes  of  all  other  pending  cases 
(252)  are  "unobserved."  These  unobserved  case  outcomes  limit  our  ability  to  estimate  the  cost  savings 
associated  with  ORT.  This  is  an  expected,  unavoidable  problem  which  was  discussed  in  detail  in  our 
Evaluation  Design.  Failure  to  take  account  of  the  censored  case  outcomes,  especially  for  cases  that 
opened  after  ORT  started,  results  in  bias  when  comparing  outcomes  observed  from  the  ORT  and  pre-ORT 
periods,  or  between  demonstration  and  comparison  states. 

We  use  two  general  strategies  for  handling  the  censored  nature  of  CIMS  data  First,  we  use  •  artificial 
censoring"  techniques  so  that  the  extent  of  censoring  in  the  pre-ORT  data  is  (artificially)  similar  to  the 


8  Nole  thai,  in  our  survey  of  cases  (see  Appendix  7).  cases  thai  span  pre-ORT  and  ORT  periods  are  termed  "hndge  cases." 

9  Note  that  for  estimaies  of  cenain  ORT  impacts,  we  consider  outcomes  from  cases  that  opened  in  (he  24  months  preceding  the  pre-ORT 
penod  (April  1 .  199 1  -  March  3 1 .  1993).   See  discussion  of  "anificial  censonng"  below. 
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ORT-period  data.  Second,  to  estimate  the  probable  estimate  of  demonstration/comparison  state  differences 
of  pending  cases,  we  estimate  the  conditional  expected  outcome  of  each  pending  case,  based  on  its  point 
in  the  case  development  process.  This  represents  our  best  guess  of  the  ultimate  outcome  of  pending  cases, 
and  fills  in  our  best  estimate  of  the  data  we  cannot  have:  data  on  the  actual  results  of  these  cases  m  future 
months.  This  section  describes  these  two  techniques  for  evaluation  of  censored  outcome  measures. 

Method  1 :  Adjusting  for  the  impact  of  censored  data  on  pre/post  analyses:  "Artificial  censoring" 
of  outcomes 

Simple  pre/post  comparisons  of  observed  case  outcomes  are  mappropnate  because  cases  that  opened  in 
the  pre-ORT  period  have  been  observed  for  a  longer  penod  of  time  than  those  from  the  demonstration 
penod.  Consequently,  the  final  outcome  for  more  cases  from  the  pre-ORT  cohort  will  be  known  (and  thus 
reported  m  CEMS),  and  the  total  receivables  recorded  m  CIMS  will  tend  to  be  higher  for  pre-ORT  cases 
than  for  cases  from  the  ORT  penod,  where  a  higher  proportion  of  cases  are  still  pending. 

In  companng  outcomes  from  the  pre-  and  post-ORT  penods,  we  must  take  account  of  the  fact  that  pre- 
ORT  cases  have  had  more  time  to  progress  through  the  case  development  cycle.  CIMS  records  the  dates 
of  vanous  milestones  in  the  case  development  process  (e.g.,  the  date  the  case  opened,  the  date  it  was 
referred  to  prosecutors,  the  date  it  went  to  trial,  or  reached  a  pre-tnal  settlement,  the  Civil  Monetary 
Penalties  Law  of  1981  (CMPL)  hearing  date,  the  sanction/exclusion  date,  the  date  the  case  closed).  CIMS 
thus  allows  us  to  perform  "slice-in-time"  analyses.  By  ignonng  outcomes  that  occurred  after  (or  before) 
a  given  date,  we  can  artificially  censor  data  from  the  pre-ORT  period  at  any  point  in  time,  to  permit 
relatively  unbiased,  direct  comparisons  to  ORT-period  data. 

The  approach  of  artificially  censoring  outcomes  based  on  the  date  that  they  occurred  combines  the  narrow 
ORTl  and  the  broad  0RT2  definitions  described  above.  Exhibit  5-3  shows  how  the  artificial  censoring 
and  double  artificial  censoring  methodologies  assign  outcomes  to  cases,  based  on  the  date  the  case  opened 
and  the  date  the  outcome  for  the  case  was  observed. 

For  a  comparison  of  "pure  ORT"  effects  (i.e.,  using  the  ORTl  definition  descnbed  above),  we  compare 
the  outcomes  observed  for  cases  that  opened  during  the  ORT  period  (i.e.,  cases  in  Box  D  of  Exhibit  5-  3) 
to  the  outcomes  observed  in  the  pre-ORT  period  for  cases  that  opened  in  that  period  (i.e.,  cases  in  box 
B).  As  a  result  of  excluding  outcomes  from  pre-ORT  cases  observed  during  ORT,  both  the  pre-ORT  and 
ORT  cohorts  have  an  identical  26  month  "window"  in  which  outcomes  can  be  observed.  Outcomes  that 
occur  after  the  window  are  not  observed;  for  cases  from  the  ORT  period  this  is  because  we  do  not  know 
what  the  outcomes  will  be  (i.e.,  cases  like  G),  and  in  the  pre-ORT  cohort  because  we  have  artificially 
censored  outcomes  from  cases  like  F,  where  the  outcome  was  not  observed  until  after  the  26-month 
window.  By  artificially  censoring  pre-ORT  penod  cases,  we  make  the  censoring  problem  identical  in  the 
pre-  and  post-ORT  periods.  In  this  way,  estimates  of  ORT  impacts  are  not  biased  by  the  greater  amount 
of  censored  data  for  ORT  cases. 

For  all  its  advantages,  however,  artificial  censoring  imposes  an  important  cost.  Specifically,  companson 
of  cases  in  Box  D  to  those  in  Box  B  does  not  capture  any  impacts  of  ORT  on  cases  that  were  pending 
when  the  demonstration  started.  This  may  result  in  an  understatement  of  the  actual  impacts  of  ORT,  since 
ORT  may  influence  the  development  and  eventual  outcomes  of  such  cases. 
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Exhibit  5-3: 

Assigning  Cases  to  the  Pre-ORT  and  ORT  Periods: 

Artificially  Censored  and  Double  Censored  Outcome  Measures 


Pre-pre-ORT  Period 

Pre-ORT  Period 

1                ORT  period 

Po 

st-ORT 

•  ♦  Cast 

Box  A:  Pending  at  start  of  pre-ORT  peric 

ORT  observation  period  ^through  5/97f 

Case  D 
Case  G 

d 

•  

•  

 ♦  Case  A 

 ♦  CaseB 

Rr»y  R-  Piirp  Prp-ORT  •  

— ^ 

l_l  \J  A  u.  rule  lie      1 1  1 

•  ♦  CaseB 

Box  C:  ORT  influenced 

•  

 ♦     Case  F 

Pre-ORT  observation  period  (through  5/95) 

«  

Box  D:  Pure-ORT 

► 

•  ♦  CaseH 

4/91  4/ 

33                                                                   4/95      5/95                                                          4/97  5/97 

•    Case  Opening  Date  ♦   Case  Outcome  Date 
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To  measure  the  impact  of  ORT  on  cases  that  were  pending  when  ORT  started  (i.e.,  the  additional  cases 
defined  as  ORT-reiated  using  the  0RT2  definition  above),  we  compared  outcomes  observed  during  the 
ORT period'°  for  cases  that  opened  in  the  pre-ORT period  (i.e..  cases  in  Box  C)  to  outcomes  observed  />i 
the  pre-ORT  period"  for  cases  that  opened  prior  to  the  pre-ORT  period''  (i.e..  cases  in  Bo.x  A). 

This  companson  uses  the  technique  of  double  artificially  censored  outcome  measures.  Outcomes  of 
cases  that  opened  in  the  pre-ORT  period  are  censored  if  they  did  not  occur  during  the  ORT  period,  smce 
ORT  would  not  influence  outcomes  observed  prior  to  the  demonstration  start  date.  Outcomes  of  cases  that 
opened  in  the  two  years  preceding  the  pre-ORT  period  (i.e..  the  pre-pre-ORT  period)  were  censored  if  they 
did  not  occur  during  the  pre-ORT  period  (i.e..  the  outcome  could  be  censored  either  because  it  was 
observed  before  the  pre-ORT  penod  or  dunng  the  demonstration  penod).  Companson  of  double 
artificially  censored  outcome  measures  permits  estimation  of  the  effects  of  ORT  on  more  "mature  "  cases, 
that  were  pending  pnor  to  the  demonstration,  by  using  pre-pre  ORT  cases  to  estimate  average  outcomes 
in  the  absence  of  ORT. 

Because  ORT  did  not  affect  the  volume  of  cases  from  the  pre-ORT  penod  that  were  pending  at  the  stan 
of  the  demonstration  period  (and  the  pre-ORT  penod  did  not  influence  the  volume  of  cases  from  the  pre- 
pre-ORT  penod  that  were  pending  at  the  start  of  the  pre-ORT  period),  censored  outcome  measures  were 
adjusted  to  reflect  volume  constant  outcomes".  For  example,  assume  that  the  pre-ORT  penod  had  no 
influence  on  the  outcome  on  cases  that  were  pending  at  the  stan  of  the  penod.  but  that  there  were  twice 
as  many  pending  cases  from  companson  states  as  from  ORT  states.  If  volume  constant  outcome  measures 
were  not  used,  the  double  artificial  censoring  technique  would  estimate  that  the  pre-ORT  penod  was 
associated  with  a  positive  impact  on  pending  cases  in  comparison  states,  simply  because  there  were  more 
pending  cases  from  comparison  states.  The  number  of  pending  cases  at  the  start  of  the  ORT  and  pre-ORT 
penods  is  exogenous,  as  the  cases  were  initiated  before  the  stan  of  the  period.  The  assumption  underlying 
this  approach  is  that  the  resources  available  to  pursue  cases  (which  presumably  influence  case  outcomes) 
do  not  vary  with  the  volume  of  cases. 

Method  2:  Adjusting  for  the  impact  of  censored  data  on  demonstration/comparison  state 
analyses:  Estimating  the  expected  outcome  of  pending  cases 

The  use  of  artificially  censored  outcome  measures  allows  for  comparison  of  outcomes  observed  in  the 
ORT  and  pre-ORT  periods.  It  does  not ,  however,  control  for  different  levels  of  censoring  that  may  be 
present  between  cases  in  demonstration  and  comparison  states.  If  ORT  affected  the  volume  of  cases,  then 
more  cases  may  be  pending  from  ORT  states  during  the  demonstration  penod.  relative  to  companson  states 
or  the  pre-ORT  penod.  In  estimating  the  impact  of  ORT  on  01  outcomes,  it  is  important  to  include  the 
expected  receivables  from  cases  that  were  still  pending  as  of  the  data  cutoff  point,  as  some  of  these  cases 
will  eventually  result  in  the  identification  of  receivables  and  produce  actual  Medicare  recovenes. 


10  Including  .Apnl  and  May  1997.  which  are  included  in  our  data. 

1 1  Including  Apnl  and  May  1995.  so  thai  txjth  the  pre-  and  post-ORT  windows  cover  the  same  length  of  time. 

12  We  included  cases  thai  opened  between  Apnl  1991  and  March  1993  m  the  pre  pre-ORT'  penod. 

13  Note  thai  because  the  number  of  pending  cases  from  the  pre-ORT  penod  was  nearly  identical  in  ORT  ( 1 09)  and  companson  states 
(107).  volume  constant  outcome  measures  were  only  applied  to  data  from  the  pre-pre-ORT  penod.  There  were  nearly  twice  as  many 
cases  from  companson  slates  at  the  start  of  the  pre-ORT  penod  as  from  ORT  states  (see  Exhibit  5-2 1 ). 
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For  comparison/demonstration  state  analyses,  the  final  outcomes  for  pending  cases  are  based  on  their 
conditional  expected  receivables.  These  figures  are  estimated  using  the  results  of  cases  opened  in  the 
pre-ORT  penod  that  reached  a  final  outcome.  We  used  all  ORT  subject-type  cases  from  ORT  and 
comparison  states  that  opened  between  1991  and  1994  to  denve  estimates  of  conditional  expected 
receivables.  (Later  cases  were  excluded  because  there  may  be  systematic  differences  between  cases  that 
proceed  more  quickly  through  the  case  development  cycle  and  other  cases.) 

Given  the  extremely  large  standard  deviation  of  estimated  receivables  identified  for  cases  (see  Exhibit 
4),  these  estimates  yield  imprecise  estimates  and  must  be  regarded  as  speculative  estimates.  There  are 
large  vanations  in  the  amount  of  receivables  and  ultimate  Medicare  recovenes  for  different  cases.  What 
this  means  is  that,  based  on  the  information  contained  in  CIMS,  it  is  difficult  to  predict  in  advance  the  final 
outcome  of  pending  cases.  Nonetheless,  the  conditional  expected  value  approach  gives  us  the  hesi 
estimate  possible,  given  the  information  observed  to  date  for  the  case.  Note  that  since  most  of  the 
information  available  on  completed  cases  is  from  pre-ORT  cases,  we  must  use  this  pre-demonstration 
penod  data  to  estimate  expected  outcomes  for  ORT  cases,  which  may  not  be  appropnate  if  ORT  has 
positive  effects  on  the  probability  and  size  of  positive  outcomes.  In  that  sense,  our  estimates  of  receivables 
based  on  this  method  are  likely  conservative. 


Exhibit  5-4: 

Conditional  Expected  Savings:  Expected  Receivables  Established  for  Cases  at  Various 
Points  in  the  Case  Development  Cycle  (Non-exclusion  pre-ORT  cases  only)  


Mean 

Std 

%  with 

Point  in  the  case  development  cycle: 

N 

outcome 

deviation 

receivables 

Case  opened  but  not  accepted  for  prosecution 

535 

$400,244 

$2,721,776 

21 .3% 

Case  opened  and  accepted  for  civil  prosecution 

42 

$1,146,356 

$2,474,196 

88.1% 

Case  opened  and  accepted  for  criminal 

79 

$685,857 

$3,078,598 

67.1% 

prosecution 


Notes;  Based  on  all  non-pending  ORT  subject  type  cases  from  ORT  or  companson  states  that  opened  between  1991  and 
1994. 

Source:  CIMS   


Calculating  Receivables  Established  and  Actual  Medicare  Recoveries 

Another  issue  of  importance  in  our  analysis  is  how  much  each  case  generates  in  actual  Medicare 
recoveries-the  "dollars  to  the  Trust  Fund"-as  opposed  to  accounts  receivable  that  may  or  may  not  yield 
returns  to  the  government.  HCFA  has  tracked  01  receivables  resulting  from  ORT  cases  through  to 
collection  of  actual  payment.  Because  no  such  data  were  available  for  either  the  pre-ORT  penod  or  for 
comparison  states,  we  were  not  able  to  estimate  pre/post  or  double  difference  estimates  of  the  impact  of 
,ORT  on  actual  Medicare  recoveries.  Our  analysis  of  recovenes  is,  therefore,  purely  descnptive.  We  used 
the  HCFA  data  to  estimate  the  proportion  of  the  receivables  established  from  OI  cases  that  were  actually 
collected  by  HCFA. 

Because  HCFA  will  continue  to  recover  funds  resulting  from  ORT  cases,  the  estimates  of  Medicare 
recoveries  presented  in  this  section  are  a  lower-bound  estimate.  (With  the  passage  of  time,  additional 
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Because  HCFA  will  continue  to  recover  funds  resulting  from  ORT  cases,  the  estimates  of  Medicare 
recovenes  presented  in  this  section  are  a  lower-bound  estimate.  (With  the  passage  of  time,  additional 
payments  will  doubtless  be  made.)  But,  while  they  are  likely  to  mcrease  with  time,  these  lower  bound 
estimates  of  "money  in  the  bank"  are  the  most  conservative  estimates  of  "ORT  effects." 

Data  on  the  receivables  resulting  from  OI  cases  is  contained  m  CIMS.  To  prevent  one  individual  case 
from  having  a  disproportionate  effect  on  impact  analyses,  one  outlier  case  (a  non-ORT  case  that  CIMS 
reported  as  resulting  in  more  than  S200  million  in  receivables,  more  than  six  times  the  total  for  the  second 
highest  case  used  in  our  impact  analyses)  was  truncated  with  receivables  established  of  S35  million.  This 
figure  was  selected  so  that  this  case  would  still  have  higher  receivables  than  any  other  case  included  in  the 
analyses,  while  preventing  it  from  having  a  disproponionate  influence  on  our  impact  analyses. 

5.2    Impact  of  ORT  on  Case  Initiation 

In  the  sections  that  follow,  our  discussion  of  the  impacts  of  ORT  is  divided  into  three  sections.  First,  we 
discuss  the  impact  of  ORT  on  case  volume,  sources,  subject  types,  and  allegations.  Second,  we  explore 
preliminary  estimates  of  ORT  on  case  development  and  case  outcomes,  using  the  artificial  censonng 
methodology  described  above.  Finally,  in  Section  5.4.3.  we  report  on  the  outcomes  observed  to  date  for 
cases  identified  as  ORT-related  in  CIMS. 

5.2.1     Impact  of  ORT  on  Case  Volume 

Non-exclusion'^  Cases 

ORT  had  a  large  impact  on  the  volume  of  non-exclusion  cases  opened  by  OI.  Across  the  five 
demonstration  states,  volume  increased  by  62%,  from  164  to  265  cases  during  ORT  (Exhibit  5-5). 
Volume  of  non-exclusion  cases  fell  in  companson  states,  from  173  to  139  cases,  a  20%  decline.  Given 
the  decrease  that  occurred  in  comparison  states,  our  best  estimate  is  that  ORT  was  associated  with  an 
103%  increase  in  the  number  of  non-exclusion  cases  opened  against  ORT  focus  area  providers.  The 
volume  of  cases  increased  in  all  five  ORT  states. 

The  impact  of  ORT  on  volume  was  consistent  throughout  the  two-year  demonstration  period  (Exhibit  5- 
6).  Dunng  every  quarter  of  ORT,  the  number  of  cases  opened  in  ORT  states  was  higher  than  the  highest 
number  of  cases  (25)  that  opened  during  any  pre-ORT  quarter.  Even  in  the  first  quarter  of  ORT,  before 
many  ORT  projects  had  begun,  volume  increased  to  27  cases,  from  an  average  of  19.5  cases  per  quarter 
in  the  year  preceding  ORT.  It  would  thus  appear  that  high-level  attention  and  mobilization  of  staff  alone 
had  some  impact  on  case  volume,  since,  in  the  early  penod.  none  of  the  ORT  resources  had  been  released. 

ORT's  impact  on  volume  was  largest  in  1996,  especially  dunng  the  first  quarter.  During  the  pre-ORT 
period,  volume  was  similar  in  ORT  (164  cases)  and  companson  states  (173  cases),  but  in  1996,  142  cases 
opened  in  demonstration  states,  compared  to  only  57  in  the  companson  states.  During  the  first  quarter  of 
1996,  53  cases  opened  across  the  five  ORT  states,  a  517c  increase  over  the  previous  quarter,  and  neady 
200%  more  than  opened  in  comparison  states  during  the  same  quarter. 


14     Non  exclusion  cases  include  those  lhat  did  not  involve  an  administrative  sanction  or  exclusion.  Non-exclusion  cases  typically  involve 
active  investigative  work  by  OI  agents.  In  contrast,  exclusion  cases  are  processed  by  OI.  but  the  work  is  normally  a  routine  follow-up  to 
findings  of  guilt  brought  about  by  the  investigative  work  of  others  Hence,  we  discuss  separately  exclusion  and  non-exclusion  cases 
recorded  in  CIMS  (see  Section  5.1.2). 
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Exhibit  5-5:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non  ORT  states  with  an  Ol  suboffice 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 


Abt  Associates  Inc. 


Appendix  5:  Analysis  of  CIMS 


5-13 


Exhibit  5-6:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  by  Quarter  -  Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 
2  Comparison  states  include  non  ORT  states  with  an  Ol  suboffice 

3.  Pre  ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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Exclusion  Cases 


While  the  number  of  exclusion  cases  increased  from  72  to  184  in  ORT  states  during  ORT  (a  156% 
increase),  ORT  was  actually  associated  with  an  87c  net  decrease  in  the  volume  of  exclusion  cases,  due  to 
an  even  larger  increase  that  occurred  in  comparison  states,  where  volume  mcreased  by  .  from  94  to 
261  cases  (Exhibit  5-7).  Between  January  and  March  1997.  95  exclusion  cases  were  processed  m 
companson  states  (Exhibit  5-8).  The  number  of  exclusion  cases  in  that  quarter  was  more  than  the  total 
across  companson  states  dunng  the  entire  pre-ORT  penod.  In  ORT  states,  only  33  exclusion  cases  were 
processed  during  this  quarter. 

5.2.2    Changes  in  Volume  by  State 

Relative  to  the  pre-ORT  period.  ORT  was  associated  with  a  higher  volume  of  both  exclusion  and  non- 
exclusion  ca.ses  in  all  five  demonstration  states.  For  non-exclusion  cases,  the  largest  increa.ses  in  volume 
occurred  in  New  York  and  Texas  (Exhibit  5-9).  With  respect  to  exclusion  cases,  Texas  and  Flonda  had 
the  largest  increases  in  volume  (Exhibit  5-10).  though,  as  noted  above,  the  volume  of  exclusion  cases 
increased  more  in  comparison  states.  Below,  we  bnefly  descnbe  the  changes  in  volume  associated  with 
ORT  individually  for  each  demonstration  state. 


California 


In  California,  the  number  of  non-exclusion  cases  increased  by  247c.  from  46  to  57  cases.  Most  of  the 
increase  was  due  to  higher  volume  of  HHA  cases,  which  nearly  doubled  from  11  to  2 1 .  The  number  of 
DME  cases  declined  slightly,  from  27  to  26  cases. 

The  number  of  exclusion  cases  in  the  state  increased  from  1 1  to  23.  a  smaller  percentage  increase  than  in 
any  ORT  state  other  than  New  York,  and  a  smaller  increase  than  occurred  across  the  comparison  states. 
The  number  of  SNF  exclusion  cases  increased  from  zero  to  7,  and  there  were  smaller  increases  in 
exclusions  of  DME  suppliers  ( 1 1  to  1 5  cases)  and  HHAs  (zero  to  1 ). 


Florida 


Florida  had  a  smaller  percentage  increase  in  non-exclusion  cases  than  any  other  ORT  state.  Volume 
increased  from  30  to  34  cases,  a  1 37o  increase.  This  increase  was  spread  across  all  three  major  ORT  focus 
areas,  with  small  increases  in  the  volume  of  cases  opened  against  DME  suppliers  (26  to  28),  HHAs  (3  to 
4),  and  SNFs  ( 1  to  2).  During  our  site  visit  discussions,  we  learned  that  01  in  Florida  was  reluctant  to 
accept  HHA  cases  (the  St.  John's  Home  Health  Agency  case  is  a  notable  exception),  a  reluctance  that  01 
attributes  to  difficulties  in  getting  the  U.S.  Attorney's  Office  in  Miami  to  prosecute  such  cases. 

In  Florida.  ORT  was  associated  with  a  large  increase  in  the  volume  of  exclusion  cases,  which  increased 
from  7  to  30.  Most  of  this  increase  was  the  result  of  additional  exclusions  of  DME  suppliers.  There  were 
25  DME  supplier  exclusion  cases  during  ORT,  compared  to  four  in  the  pre-ORT  penod. 

Illinois 

During  both  the  pre-  and  post-ORT  periods.  Illinois  had  the  smallest  number  of  both  exclusion  and  non- 
e.xclusion  cases.  The  volume  of  non-exclusion  cases  increased  from  10  to  15  dunng  ORT.  Most  of  the 
increase  was  attnbutable  to  a  50%  increase  in  cases  against  DME  suppliers  (10  to  15  cases).  In  Illinois, 
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Exhibit  5-7:  ORT  Impact  Analyses: 
Volume  of  Cases  Opened  -  Exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:      1 .  Includes  ORT  sub|ect  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non-ORT  slates  witti  an  Ol  subolfice 

3.  Pre  ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95  3/97 
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Exhibit  5-8:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  by  Quarter  -  Exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:      1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non  ORT  slates  with  an  Ol  subotfice. 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95  Post  ORT  period  includes  cases  opened  between  4/95  3,97 
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Exhibit  5-9:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  by  states  -  Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:      1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non-ORT  states  with  an  Ol  subotfice 

3.  Pre  ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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Exhibit  5-10:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  by  states  -  Exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28.  1997) 

Notes:      1  Includes  ORT  sub|ect  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 

2  Comparison  states  include  non  ORT  states  with  an  Ol  subotfice 

3  Pre-ORT  period  includes  cases  opened  between  4/93-3/95  Post  ORT  period  includes  cases  opened  between  4/95  3/97 
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ORT  was  associated  with  a  decrease  in  the  number  of  HHA  cases,  and  there  were  only  2  SNF  cases 
opened  dunng  ORT  in  the  state. 

The  number  of  Illinois  exclusion  cases  increased  from  4  to  11  dunng  the  demonstration  period.  The 
increase  was  due  to  an  increase  in  SNF  exclusions,  which  increased  from  zero  to  7  during  ORT. 


New  York 


The  volume  of  non-exclusion  cases  in  New  York  more  than  doubled  during  ORT,  from  53  to  !  1 1  cases. 
There  were  large  increases  in  the  volume  of  cases  for  all  three  major  ORT  focus  areas.  Fifty-one  of  the 
53  pre-ORT  cases  involved  DME  suppliers,  and  the  number  of  DME  supplier  cases  increased  to  86  dunng 
the  demonstration  penod.  As  a  result  of  ORT,  01  investigators  in  New  York  broadened  their  focus 
beyond  investigations  of  DME  suppliers.  The  number  of  coses  opened  against  HHAs  increased  from  zero 
to  1 3,  and  the  volume  of  SNF  cases  increased  from  2  to  1 2. 

With  the  exception  of  SNFs,  ORT  was  associated  with  little  change  in  the  number  of  exclusions  processed 
by  New  York.  The  number  of  SNF  exclusions  increased  from  2  to  15.  The  number  of  HHA  exclusions 
actually  fell,  from  5  to  2. 


Texas 


In  Texas,  the  number  of  non-exclusion  cases  nearly  doubled,  increasing  from  25  to  48.  The  increase  was 
the  result  of  a  dramatic  increase  in  HHA  cases,  which  increased  by  nearly  200%  during  ORT,  from  1 1  to 
32  cases.  The  number  of  DME  supplier  cases  did  not  change  dunng  ORT,  and  there  were  only  three  non- 
exclusion  SNF  cases  in  Texas  during  the  demonstration  period. 

The  number  of  Texas  exclusion  cases  increased  by  more  than  300%  during  ORT,  from  20  to  81.  The 
increase  was  due  primarily  due  a  large  increase  in  the  number  of  SNF  exclusions.  There  were  65 
exclusions  of  Texas  nursing  homes  during  ORT,  compared  to  only  15  in  the  pre-ORT  period.  In  the  pre- 
ORT  penod,  Texas  was  responsible  for  28%  of  the  exclusion  cases  processed  across  the  five 
demonstration  states.  This  proportion  increased  to  44 7o  during  ORT. 

5.2.3    Estimate  of  Possible  Spillover  and  Diversion  of  ORT  Effects 

Comparison  of  Volume  in  ORT  and  Non-ORT  Regions 

The  number  of  non-exclusion  cases  in  non-ORT  states  within  ORT  regions  decreased  from  92  to  75  during 
ORT.  an  1 8%)  decrease.  In  comparison  states  in  non-ORT  regions,  the  volume  of  cases  fell  by  21%  during 
the  demonstration  penod,  from  81  to  64  cases  (Exhibit  5-11).  This  result  suggests  that  the  increase  in 
volume  observed  in  ORT  states  was  not  simply  the  result  of  diversion  of  resources  from  non-ORT  states 
within  ORT  regions.  As  a  result,  the  impacts  reported  throughout  this  appendix  represent  real  effects 
rather  than  simply  the  reallocation  of  01  resources. 

The  number  of  non-exclusion  cases  in  non-ORT  regions  increased  from  23  to  107  dunng  the 
demonstration  penod  (Exhibit  5-12).  In  comparison  states  within  ORT  regions,  the  number  of  exclusion 
cases  increased,  but  by  a  much  smaller  proportion,  from  71  to  154  cases.  The  large  increase  in  non-ORT 
regions  was  due  principally  to  a  large  number  of  exclusion  cases  handled  by  01  in  Aricansas  and  Washing- 
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Exhibit  5-11:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  in  ORT  and  non-ORT  Regions 
Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 
2  Comparison  stales  include  non-ORT  states  witti  an  Ol  subotfice. 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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Exhibit  5-12:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  in  ORT  and  non-ORT  Regions 
Exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28.  1997) 

Notes:      1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non-ORT  slates  with  an  Ol  subotfice 

3.  Pre  ORT  penod  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97. 
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ton.  Given  the  nature  of  exclusion  cases,  we  do  not  beUeve  that  the  difference  in  exclusion  cases  for 
comparison  states  m  ORT  and  non-ORT  regions  is  related  to  any  diversion  of  resources. 

Comparison  of  Volume  in  ORT  States  and  Non-ORT  States  within  ORT  Regions 

The  volume  of  non-exclusion  cases  opened  in  non-ORT  states  withm  ORT  regions  declined  from  92  to 
75  dunng  ORT.  an  18%  decrease  (Exhibit  5-13).  This  decrease  was  very  similar  to  that  observed  m 
companson  states  outside  of  ORT  regions  (from  Exhibit  5-14),  suggesting  that,  if  there  were  any  leakage 
of  ORT  effects  into  other  states  or  diversion  of  efforts  to  ORT  stales,  that  the  two  effects  more  or  less 
offset  each  other. 

Because  we  found  no  evidence  that  ORT  resources  had  been  diverted  to  pursue  efforts  in  demonstration 
states,  we  did  not  compare  outcomes  for  ORT  states  to  non-ORT  states  within  ORT  regions  for  the 
outcome  measures  analyzed  later  in  this  appendix. 

5.2.4     Impact  of  ORT  on  Subject  Types" 


Non-exclusion  Cases 


ORT  was  associated  with  a  large  increase  in  volume  across  all  three  major  ORT  focus  areas: 

•  In  ORT  states,  the  number  of  home  health  agency  cases  increased  from  29  to  72  dunng  the 
demonstration  period,  an  increase  of  148%.  The  volume  of  home  health  agency  cases  declined 
slightly  in  comparison  states,  and.  taking  into  account  the  pattern  observed  in  companson  states. 
ORT  was  associated  with  a  167%  increase  in  home  health  agency  cases. 

•  The  number  of  DME  supplier  cases  increased  by  33%  in  ORT  states  (Exhibit  5-15),  from  123  to 
163  cases.  Combined  with  the  46%  decrease  in  DME  supplier  cases  observed  in  comparison 
states.  ORT  was  associated  with  a  146%  increase  in  cases  opened  against  DME  suppliers. 

•  Investigations  of  nureing  homes  seems  to  have  become  a  higher  priority  for  01  nationwide  dunng 
the  demonstration  period  (not  only  in  ORT  states),  as  there  were  large  increases  in  the  volume  of 
nursing  home  cases  in  both  ORT  and  comparison  states.  The  number  of  nursing  home  cases  more 
than  doubled  in  comparison  states,  from  16  to  34  cases.  A  much  larger  increase  in  volume 
occurred  in  ORT  states,  where  the  number  of  nursing  home  cases  increased  by  164%,  from  1 1  to 
29  cases.  Taking  nationwide  patterns  into  account,  the  double  difference  estimator  suggests  that 
ORT  resulted  in  a  46%  increase  in  nursing  home  cases. 


Exclusion  Cases 

We  noted  above  a  large  increase  in  exclusion  cases  observed  in  comparison  states,  especially  during 
last  six  months  of  the  demonstration  period.  Despite  the  increase  that  occun-ed  in  comparison  states 


15     -n,e  subject  type  field  .n  CIMS  descnbes  the  5ubject(s)  of  the  tnvestigalion  (i.e..  whether  they  an:  a  DME  supplier,  a  home  health 
agency,  a  nursing  home  etc.). 
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Washington.  Given  the  namre  of  exclusion  cases,  we  do  not  believe  that  the  difference  in  exclusion  cases 
for  comparison  states  in  ORT  and  non-ORT  regions  is  related  to  any  diversion  of  resources. 


Comparison  of  Volume  in  ORT  States  and  Non-ORT  States  within  ORT  Regions 

The  volume  of  non-exclusion  cases  opened  in  non-ORT  states  within  ORT  regions  declined  from  92  to 
75  dunng  ORT.  an  18%  decrease  (Exhibit  5-13).  This  decrease  was  very  similar  to  that  observed  m 
comparison  states  outside  of  ORT  regions  (from  Exhibit  5-14),  suggesting  that,  if  there  were  any  leakage 
of  ORT  effects  into  other  states  or  diversion  of  efforts  to  ORT  states,  that  the  two  effects  more  or  less 
offset  each  other. 

Because  we  found  no  evidence  that  ORT  resources  had  been  diverted  to  pursue  efforts  in  demonstration 
slates,  we  did  not  compare  outcomes  for  ORT  states  to  non-ORT  states  within  ORT  regions  for  the 
outcome  measures  analyzed  later  in  this  appendix. 

5.2.4     Impact  of  ORT  on  Subject  Types'^ 

Non-exclusion  Cases 

ORT  was  associated  with  a  large  increase  in  volume  across  all  three  major  ORT  focus  areas: 

In  ORT  states,  the  number  of  home  health  agency  cases  increased  from  29  to  72  during  the 
demonstration  period,  an  increase  of  1 48%.  The  volume  of  home  health  agency  cases  declined 
slightly  in  comparison  states,  and,  taking  into  account  the  pattern  observed  in  comparison  states, 
ORT  was  associated  with  a  J 677c  increase  in  home  health  agency  cases. 

•  The  number  of  DME  supplier  cases  increased  by  33%  in  ORT  states  (Exhibit  5-15),  from  1 23  to 
163  cases.  Combined  with  the  46%  decrease  in  DME  supplier  cases  observed  in  comparison 
states.  ORT  was  associated  with  a  146%  increase  in  cases  opened  agamst  DME  suppliers. 

*  Investigations  of  nursing  homes  seems  to  have  become  a  higher  priority  for  01  nationwide  during 
the  demonstration  period  (not  only  in  ORT  states),  as  there  were  large  increases  in  the  volume  of 
nursing  home  cases  in  both  ORT  and  comparison  states.  The  number  of  nursing  home  cases  more 
than  doubled  in  comparison  states,  from  16  to  34  cases.  A  much  larger  increase  in  volume 
occurred  in  ORT  states,  where  the  number  of  nursing  home  cases  increased  by  164%,  from  1 1  to 
29  cases.  Taking  nationwide  patterns  into  account,  the  double  difference  estimator  suggests  that 
ORT  resulted  in  a  46%  increase  in  nursing  home  cases. 

Exclusion  Cases 

We  noted  above  a  large  increase  in  exclusion  cases  observed  in  comparison  states,  especially  during  the 
last  six  months  of  the  demonstration  period.  Despite  the  increase  that  occurred  in  comparison  states,  in 


1 5    The  subject  type  field  in  CIMS  descnbes  the  subject(s)  of  the  investigation  (i.e..  whether  they  are  a  DME  supplier,  a  home  health 
agency,  a  nursing  home  etc.). 
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Exhibit  5-13:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  within  ORT  regions 
Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  o(  May  28,  1997) 

Notes:     1 .  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non-ORT  stales  with  an  Ol  suboftice 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95  3/97 
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Exhibit  5-14:  ORT  Impact  Analyses 
Volume  Opened  within  ORT  Regions  -  Exclusion  Cases  Only 
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3.  Pre  ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97. 
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Exhibit  5-15:  ORT  Impact  Analyses 

Impact  of  ORT  on  Volume:  Cases  Opened  by  Subject  Type-  Non-exclusion  Cases 
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the  double  difference  model,  ORT  was  associated  with  moderate  increases  in  the  volume  of  DME 
supplier  and  nursing  home  exclusion  cases. 

Across  ORT  states,  the  number  of  DME  supplier  exclusion  cases  increased  by  62%,  from  43  to  70  cases 
(Exhibit  5-16).  In  comparison  states,  the  volume  of  DME  supplier  exclusion  cases  increased  by  34%,  from 
29  to  39  cases.  ORT  was  thus  associated  with  a  21 7c  increase  in  the  volume  of  exclusion  cases  against 
DME  suppliers. 


Exhibit  5-16:  ORT  Impact  Analyses 

Impact  of  ORT  on  Volume:  Cases  Opened  by  Subject  Type-  Exclusion  Cases 
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Due  primarily  to  higher  volume  in  Texas,  the  number  of  nursing  home  exclusion  cases  increased  by  more 
than  400%,  from  1 8  to  96  cases.  The  number  of  nursing  home  exclusion  cases  increased  by  228%,  from 
61  to  200  cases,  across  comparison  states.  There  were  fewer  home  health  agency  exclusion  cases  in  both 
ORT  and  comparison  states.  In  ORT  states,  the  number  of  home  health  agency  exclusion  cases  increased 
from  1 1  to  1 8,  a  smaller  increase  than  occurred  in  comparison  states,  where  the  number  of  exclusion  cases 
increased  from  4  to  21  cases,  a  425%  increase. 
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5.2.5  Impact  of  ORT  on  Case  Sources 


Non-exclusion  Cases 

The  ORT  partnership  mcxiel  was  associated  with  an  increase  in  the  number  of  referrals  from  ORT  partners 
and  other  agencies  that  participated  in  ORT.  Given  the  large  increases  in  case  volume  associated  with 
ORT.  the  actual  proportion  of  cases  referred  by  these  agencies  did  not  increase,  although  the  absolute 
number  of  referrals  did  rise. 

The  number  of  cases  referred  by  HHS  agencies  other  than  HCFA  (pnmanly  OAS)  increased  by 
47%  in  ORT  states,  while  declining  slightly  in  companson  states  (Exhibit  5-17). 

The  small  number  of  cases  referred  directly  by  HCFA  was  small,  but  increased  from  5  to  8  across 
ORT  stales  (remainmg  at  about  3%  of  total  cases),  but  declined  from  6  to  2  in  companson  states. 

The  number  of  cases  in  demonstration  states  dunng  ORT  attributable  to  a  hotline  call  was 
relatively  small  (13),  but  this  represented  a  large  increase  relative  to  the  pre-ORT  penod,  when 
only  2  cases  could  be  traced  to  hotline  calls.  No  increase  in  cases  referred  through  hotline  calls 
occurred  in  comparison  states.  Six  of  the  13  cases  that  resulted  from  a  hotline  call  were  from 
California.  New  York  had  three  cases  that  opened  following  a  hotline  call,  while  Flonda  and 
Texas  had  two  each. 

•  Although  the  proportion  of  cases  referred  by  HCFA  contractors  increased  in  ORT  states  from  53 
to  81  cases,  there  was  little  change  in  the  volume  of  cases  referred  by  HCFA  contractors  in 
comparison  states.  Texas  and  Florida  had  especially  large  increases  in  referrals  from  HCFA 
contractors.  In  Florida,  referrals  from  HCFA  contractors  increased  from  9  to  17  during  the 
demonstration  period.  HCFA  contractor  referrals  increased  from  3  to  9  in  Texas. 

During  ORT,  there  was  a  large  increase  in  the  volume  of  cases  referred  by  state  agencies.'*  The 
volume  of  cases  referred  by  state  agencies  increased  by  150%  in  ORT  states  (28  to  70  cases),  but 
declined  from  24  to  16  across  comparison  states.  Much  of  the  increase  was  due  to  higher  volume 
in  New  York,  where  referrals  from  state  agencies  increased  from  19  to  48  dunng  ORT.  In  Texas, 
the  number  of  referrals  from  state  agencies  increased  from  5  to  14. 

•  The  number  of  cases  referred  by  Medicare  beneficiaries,  private  citizens,  and  employees  of 
providers  accused  of  fraud  increased  by  more  than  100%  during  the  demonstration  period  across 
the  five  demonstration  states  (increasing  from  13%;  to  17%  of  total  referrals).  (Refenals  from 
these  sources  in  comparison  states  actually  decreased  dunng  the  ORT  period.)  The  increase  in 
ORT  states  was  due  largely  to  higher  volume  in  Texas  and  New  York.  In  Texas,  the  number  of 
cases  referred  by  beneficiaries,  private  citizens,  or  employees  increased  from  3  to  1 6.  In  New 
York,  ORT-period  cases  from  this  source  increased  from  5  to  12.  While  it  is  not  possible  to 
establish  a  direct  causal  link  between  these  increases  and  ORT  outreach  efforts,  this  increased 


1 6    CIMS  does  not  identify  the  name  of  the  state  or  local  agency  that  referred  the  case.  These  cases  may  reHect  referrals  from  LTC 
Surveyors.  Ombudsmen,  and  others. 
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Exhibit  5-17:  ORT  Impact  Analyses 


Impact  of  ORT  on  Volume:  Cases  Opened  by  Source-  Non-exclusion  Cases 
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Notes:     Hotline  calls  include:  a)  all  cases  with  a  Hotline  identification  number  recorded  in  CIMS,  or  b)  a  source 
code  of  "Federal  government  hotline"  or  "Federal  HHS  Hotline"    State  agencies  include  Medicaid  Fraud 
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codes  include  the  following  sub-source  fields:  HHS/Office  of  General  Counsel,  Dept.  of  HHS,  Audit,  and 
Dept  of  HHS/OIG  and  the  "Audif  major  source  field. 


Source:  CIMS 
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volume  of  referrals  may  reflect  the  results  of  outreach  efforts  targeted  at  Medicare  beneficiaries,  providers, 
trade  associations,  and  ombudsmen. 

Overall,  there  were  increases  in  referrals  from  all  sources,  especially  state  agencies,  beneficianes/ 
employees/anonymous,  and  HCFA  Contractors.  See  .Appendix  7  (Survey  of  Cases)  for  additional 
information  on  state  referral  sources. 

Exclusion  Cases 

Across  both  ORT  and  comparison  states,  most  exclusion  cases  resulted  from  efforts  of  a  state  agency,  and 
a  large  increase  in  exclusion  referrals  from  state  agencies  occurred  in  both  demonstration  and  companson 
states  (Exhibit  5-18).  Dunng  ORT,  there  was  also  a  large  mcrease  in  exclusion  cases  referred  by  HHS 
agencies.  In  ORT  states,  the  number  of  exclusion  cases  referred  by  HHS  increased  from  16  to  40.  This 
increa.se  was  the  result  of  additional  HHS  exclusion  referrals  in  Flonda,  where  volume  increased  from  4 
to  27  dunng  ORT. 


Exhibit  5-18  :  ORT  Impact  Analyses 

Impact  of  ORT  on  Volume:  Cases  Opened  by  Source-Exclusion  Cases  Only 
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5.2.6    Impact  of  ORT  on  Type  of  Cases  Pursued:  Major  Allegation 

The  CIMS  Situation  Report  records  the  allegations  being  pursued  in  connection  with  each  case,  allowing 
measurement  of  changes  in  the  type  of  cases  pursued  by  OI  during  ORT. 

Non-exclusion  Cases 

Changes  in  the  type  of  allegation  associated  with  ORT  include: 

•  Cases  with  allegations  related  to  bribes  or  kickbacks'^  increased  from  20  in  the  pre-ORT  penod 
to  37  dunng  ORT  (Exhibit  5-19).  In  New  York,  the  volume  of  bnbe/kickback  cases  mcreased 
from  5  to  22.  Cases  with  this  type  of  allegation  decreased  in  companson  states. 

•  The  number  of  non-exclusion  cases  that  involved  a  sanction  allegation'*  increased  by  more  than 
300%  in  ORT  states,  from  12  to  49  cases.  Cases  involving  a  sanction  allegation  increased  by  a 
smaller  amount  in  companson  states  (2  to  10  cases).  In  both  the  pre-  and  post-ORT  penod, 
almost  all  of  the  sanction  cases  onginated  in  New  York. 

The  number  of  false  billing  cases"  increased  by  44%  in  ORT  states,  while  declining  by  nearly 
40%  in  comparison  states  (95  to  58  cases).  All  ORT  states  except  for  New  York  had  an  increase 
in  the  volume  of  false  billing  cases.  The  largest  increase  occurred  in  Texas,  where  false  billing 
cases  increased  from  14  to  36  during  ORT. 

Exclusion  Cases  > 

Most  exclusion  cases  involved  one  of  the  sanction  allegations  recorded  in  CIMS,  and,  reflecting  the  large 
increase  in  exclusion  cases  processed  by  01  during  the  ORT  period,  the  volume  of  cases  with  an  exclusion 
allegation  increased  in  both  ORT  and  comparison  states.  Across  ORT  states,  the  number  of  exclusion 
cases  with  a  sanction  allegation  nearly  doubled,  from  51  to  100  cases.  A  larger  increase  occurred  in 
comparison  states,  where  the  volume  of  cases  with  a  sanction  allegation  increased  from  38  to  85. 

During  ORT,  there  was  a  large  increase  in  the  number  of  exclusion  cases  related  to  quality  of  care  issues. 
Some  of  these  cases  may  have  resulted  from  the  HCFA  Survey  and  Certification  reviews  that  were 
performed  in  all  ORT  states  during  the  demonstration  period.  The  volume  of  quality  of  care  exclusion 
cases  in  ORT  states  increased  from  4  to  34,  a  750%  increase  (Exhibit  5-20).  More  than  90%  of  the  ORT- 
period  cases  were  from  Texas.  In  comparison  states,  the  number  of  quality  of  care  exclusion  cases 
increased  from  34  to  1 10,  an  increase  of  more  than  200%. 


1 7  These  cases  include  those  with  an  allegation  of  "accepting  bribes,"  "anempting  to  bnbe. "  "standards  of  conduct."  "kickbacks."  "conllici 
of  interest."  and  "bid  rigging/pnce  fixing." 

1 8  These  cases  included  those  with  an  allegation  of  "Sanction-disciplinary  action.".  "Sanction-  Medicaid  conviction",  and  "Sanction- 
Conviction  by  HHS/OIG/OI"  These  cases  did  not.  however,  result  in  any  exclusion  recorded  in  CIMS. 

1 9  These  cases  included  those  with  any  of  the  following  allegations:  Health  care  bill  fraud,  false  stalement  in  bid  application,  false  claims 
for  pay  benefits,  duplicate  billing,  medically  unnecessary  services,  and  overcharging. 
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Exhibit  5-19:  ORT  Impact  Analyses 
Volume  of  Cases  Opened  by  Major  Allegation 
Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1 .  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non-ORT  states  with  an  Ol  subotfice. 

3.  Pre-ORT  period  Includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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Exhibit  5-20:  ORT  Impact  Analysis 
Volume  of  Cases  Opened  by  Major  Allegation 
Exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Home  and  Hospice) 

2.  Comparison  stales  include  non-ORT  states  with  an  Ol  subotfice 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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5.3    Impact  of  ORT  on  Case  Development  and  Outcomes 


Exhibits  5-21  -  5-24  report  preliminary  estimates  of  ORT  impacts  on  case  development  and  outcomes. 
These  measures  are  based  on  outcome  measures  that  have  been  artificially  censored  or  double  artificially 
censored,  depending  on  when  the  case  opened  and  when  outcomes  were  observed  for  the  case  (see  Section 
5.1  above).  Given  the  large,  often  unpredictable  amounts  that  result  from  cases  that  are  accepted  for 
prosecution  and  go  to  trial  (or  have  a  pre-trial  settlement),  our  estimates  of  ORT  impacts  may  change 
dramatically  as  pending  cases  reach  a  final  outcome.  Nevertheless,  the  figures  reported  in  this  section 
represent  our  best  estimate  of  the  impact  of  ORT  on  case  development  and  outcomes,  given  the  data  that 
are  currently  available. 

The  information  reported  in  these  exhibits  allow  for  several  estimates  of  ORT  effects: 

•  To  estimate  the  impact  of  ORT  using  the  narrow  ORT!  definition,  which  only  includes  as  ORT- 
related  those  cases  that  opened  dunng  the  demonstration  penod,  compare  the  outcomes  observed 
in  demonstration  and  comparison  states  dunng  ORT  (i.e.,  the  "pure  ORT"  box  in  Exhibit  5-3)  to 
outcomes  observed  in  the  Apnl  1993-March  1995  pre-ORT  penod  (i.e.,  the  "pure  pre-ORT"  box 
in  Exhibit  5-3). 

•  To  consider  the  potential  impact  of  ORT  on  cases  that  were  pending  when  the  demonstration 
started,  using  the  broader  0RT2  definition,  compare  the  outcomes  observed  during  ORT  for  cases 
that  opened  in  either  the  ORT  or  pre-ORT  penods  (i.e.,  the  "pure  ORT"  and  "ORT-influenced" 
boxes  from  Exhibit  5-  3)  to  outcomes  observed  dunng  the  pre-ORT  period  for  cases  that  opened 
between  April  1991-  March  1995  (i.e.,  the  "pure  pre-ORT'  and  the  "pending  at  start  of  pre-ORT 
period"  boxes  in  Exhibit  5-3). 

5.3.1  Impact  of  ORT  on  Case  Development 

Across  the  ORT  states,  68%  of  the  265  non-exclusion  cases  that  opened  during  the  demonstration  period 
were  still  pending  as  of  the  last  date  covered  by  ourCIMS  data  (May  27,  1997)  (Exhibit  5-21),  including 
21  cases  that  have  been  accepted  for  criminal  prosecution  and  four  that  were  accepted  for  civil  action. 
In  comparison  states,  about  50%  of  non-exclusion  cases  that  opened  during  the  demonstration  penod  were 
still  pending,  including  ten  cases  accepted  for  criminal  prosecution  and  ten  for  civil  action.  This  difference 
may  reflect  a  slight  "backing  up"  in  the  case  queue  in  ORT  states,  due  to  the  higher  volume  of  cases. 

Exhibit  5-21  reports  changes  associated  with  ORT  in  the  number  of  cases  successfully  referred  by  OI  for 
prosecution.  Changes  associated  with  ORT  for  cases  that  opened  dunng  the  demonstration  penod  (i.e.. 
"pure  ORT"  cases)  included  the  following: 

ORT  was  associated  with  a  modest  increase  in  the  number  of  cases  accepted  for  prosecution.  In 
demonstration  states,  the  number  of  cases  accepted  for  prosecution  increased  from  41  to  43  (using 
artificially  censored  data  for  the  pre-ORT  period).  Across  comparison  states,  the  number  of  cases 
successftiUy  refen-ed  for  prosecution  declined  from  32  to  27.  The  net  result  is  a  20%  increase  in 
the  number  of  cases  accepted  for  prosecution.  Because  there  are  many  more  pending  cases  from 
the  ORT  period  and  ORT  states  than  in  either  comparison  states  or  the  pre-ORT  period  (as  of  the 
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Exhibit  5-21:  ORT  Impact  Analyses 

Impact  of  ORT  on  Case  Development:  Non-Exclusion  Cases 


ORT  states 

Comparison  states 

date  outcome  observed: 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

date  case  opened: 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 
4/93 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 
4/95 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 
4/93 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 
4/95 

Cases  opened 

164 

(98) 

265 

173 

(197) 

139 

Cases  closed 

55 

45 

83 

69 

63 

40.5* 

69 

77 

Cases  opened  but  pending 

109 

12 

182 

40 

107 

8* 

70 

33 

%  of  open  cases  that  were 
pending 

66.5% 

68.6% 

24.3% 

61.8% 

50.3% 

19.1% 

Total  trial/pre-trial  diversions 

6 

6 

23 

17 

13 

12* 

14 

18 

Criminal 

1 

4 

5 

9 

5 

65* 

2 

5 

Civil 

5 

2 

18 

8 

8 

5.5* 

12 

13 

Total  cases  accepted  for 
prosecution 

41 

6 

43 

5 

32 

4.5* 

27 

5 

Criminal 

27 

5 

26 

5 

18 

4* 

13 

1 

Civil 

14 

1 

17 

0 

14 

0.5* 

14 

4 

Total  criminal  indictments 

1 

2 

7 

2 

2 

1* 

1 

2 

Note:       Case  development  dates  are  based  on  date  of  criminal  trial/pre-trial  settlement  for  cnminal  and  civil  cases  II  this  dale  is  not  recorded,  then  outcome  is  based  on  the 
date  that  the  case  closed. 

A;  Comparison  pehod  pre-ORT  Influenced  outcomes  have  been  adjusted  to  control  for  differences  in  the  number  of  cases  that  were  pending  at  Ihe  start  of  the  pre  ORT 
period  in  demonstration  and  comparison  states. 
Source:  CIIVIS   
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point  at  which  pre-ORT  period  outcomes  were  censored  for  pre-post  compansons),  the  impact  of 
ORT  on  cases  accepted  for  prosecution  is  likely  to  increase. 

•  ORT  was  associated  with  an  increase  in  the  number  of  cases  that  either  went  to  trial  or  reached 
a  pre-tnal  settlement.  In  the  five  ORT  states,  23  cases  from  the  demonstration  period  had  a  tnal 
or  pre-trial  diversion.  Only  six  cases  from  the  pre-ORT  penod  went  to  trial  prior  to  the  end  of  the 
artificially  censored  pre-ORT  observation  penod  ending  May  28  1995.  In  companson  states, 
there  was  little  change  in  the  number  of  cases  that  went  to  tnal  (an  increase  from  1 3  to  1 4  cases 
dunng  the  ORT  penod).  The  large  impact  of  ORT  on  the  volume  of  cases  that  went  to  inai 
suggests  that  USAOs  are  playing  a  more  active  role  m  pursuing  health  care  fraud  cases  m  ORT 
states.  It  is  also  consistent  with  01  providing  higher  quality  cases  to  prosecutors  sooner  m  the  case 
timeline. 

ORT  may  also  affect  the  development  of  cases  that  opened  in  the  pre-ORT  period  but  were  still  pending 
as  of  the  demonstration  start  date.  To  measure  the  impact  of  ORT  on  these  cases,  we  compared  case 
development  milestones  that  occurred  dunng  ORT  for  pre-ORT  cases  (i.e..  Box  B  in  Exhibit  5-2)  to 
milestones  observed  dunng  the  pre-ORT  penod  for  cases  that  opened  in  the  pre-pre-ORT  penod  (i.e..  the 
two  years  preceding  the  pre-ORT  period,  or  Box  D  in  Exhibit  5-2).  If  ORT  affects  the  development  of 
pending  cases,  then  more  of  these  cases  should  be  accepted  for  prosecution  or  go  to  trial  during  the  ORT 
penod  in  ORT  states,  relative  to  the  pre-ORT  penod  and  companson  states. 

•  ORT  did  not  appeared  to  influence  the  successful  referral  for  prosecution  for  cases  that  were 
pending  when  the  demonstration  started.  Five  cases  that  opened  in  the  pre-ORT  penod  were 
accepted  for  prosecution  during  ORT  in  the  demonstration  slates,  but  six  cases  that  opened  in  the 
two  years  prior  to  the  pre-ORT  period  were  accepted  for  prosecution  during  the  pre-ORT  period. 
There  was  little  change  in  successful  referral  for  prosecution  of  pending  cases  in  comparison 
states. 

•  During  ORT,  17  cases  that  opened  in  the  pre-ORT  period  had  either  a  trial  or  a  pre-trial 
settlement.  This  represented  a  large  increase  relative  to  the  pre-ORT  period,  when  six  cases  that 
opened  in  the  two  years  preceding  the  pre-ORT  penod  went  to  trial.  In  comparison  states,  the 
number  of  pending  cases  that  had  a  trial  or  pre-trial  settlement  increased  from  12  to  18  during  the 
demonstration  period,  but,  since  the  increase  was  larger  in  ORT  states,  ORT  was  associated  with 
an  estimated  net  increase  in  the  number  of  cases  that  went  to  trial  or  a  pre-trial  diversion. 

In  describing  the  effects  of  ORT  on  case  development,  it  must  be  kept  in  mind  that  we  have  made  no 
attempt  to  control  for  case  complexity-'  or  the  length  of  time  that  cases  were  actually  open  within  the 
observation  periods.  From  Exhibit  5-7,  most  of  the  ORT  cases  did  not  open  until  1996.  However,  the 
median  length  of  time  that  cases  were  observed  was  similar  for  cases  from  the  ORT  and  (artificially 
censored)  pre-ORT  periods,  as  well  as  from  demonstration  and  comparison  states.  This  suggests  that 
results  should  not  be  affected  by  a  skewed  distribution  of  case  opening  dates  from  the  ORT  period/ORT 
state  cohort. 


20     Nationwide,  209c  of  the  non-exclusion  cases  thai  opened  between  1 99 1  and  1994  resulted  in  successful  referral  for  prosecution.  To 
date,  only  16%  of  the  non-exclusion  cases  that  opened  dunng  the  ORT  penod  have  yet  resulted  in  successful  refeaal  to  prosecutors. 

11     Indeed,  we  were  told  by  OI  that  no  good  proxies  for  case  complexity  are  reported  in  CIMS  Our  own  work  confirms  that  point. 
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5.3.2    Impact  of  ORT  on  Receivables  Established  and  Medicare  Recoveries 

01  cases  result  in  potential  savings  for  the  Medicare  program  through  direct  monetary  recoveries  from 
providers  (i.e.,  due  to  fines,  judgements,  recovenes,  settlements,  or  restitution).  CIMS  repons  the  accounts 
receivable  that  are  established  as  a  result  of  01  cases.  It  does  not,  however,  track  these  receivables  to 
actual  collection  from  providers,  as  collection  of  these  funds  is  not  the  responsibility  of  01.  Exhibit  5-22 
reports  the  impact  of  ORT  on  receivables  established  from  01  cases. 

The  receivables  estimates  reported  here  must  be  regarded  as  preliminary,  given  that  68'7f  of  ORT  non- 
exclusion  cases  from  demonstration  states  were  still  pending  as  of  the  data  cutoff  point.  As  descnbed  in 
the  Evaluation  Design  section,  we  estimate  the  expected  receivables  of  pending  cases,  conditional  on  how 
far  the  ca^e  has  proceeded  in  the  case  development  cycle.  The  estimates  derived  using  this  method  give 
our  best  possible  guess  of  the  ultimate  outcome  of  pending  cases,  but,  must  be  regarded  as  speculative  in 
nature. 

We  report  a  variety  of  estimates  of  the  impacts  of  ORT  on  receivables  established  and  Medicare 
recoveries.  Comparisons  can  be  made  using  only  cases  which  opened  during  ORT  (i.e.,  using  the  ORTl 
definition),  or  also  including  cases  that  were  pending  when  ORT  started  that  may  have  been  influenced 
by  ORT  (i.e..  the  additional  cases  considered  to  be  ORT-related  using  the  broader  0RT2  definition).  In 
addition,  we  report  separately  three  measures  of  ORT's  impact,  which  vary  with  respect  to  what  is 
included  in  the  estimated  impact: 

•  Medicare  recoveries:  HCFA  data  on  actual  recoveries  for  ORT  cases  was  used  to  calculate  the 
percentage  of  receivables  that  were  actually  returned  to  the  Trust  Fund.  This  figure  represents 
actual  "money  in  the  bank"  and  represents  the  most  conservative  estimate  of  ORT  effects. 

•  Receivables  established:  This  measure  uses  only  receivables  reported  in  CIMS  (i.e.,  on  completed 
cases.  This  estimate  of  ORT  savings  does  not  consider  the  proportion  of  receivables  that  are  never 
collected  by  Medicare,  but  also  does  not  include  any  expected  receivables  from  pending  cases. 
As  a  result,  this  measure  reflects  a  medium-sized  estimate  of  the  impacts  of  ORT. 

Receivables  established  including  expected  receivables  for  pending  cases:  This  is  the  broadest 
estimate  of  savings  associated  with  ORT.  In  addition  to  all  of  the  receivables  identified  in  CIMS, 
it  includes  the  conditional  expected  value  of  pending  cases  (see  Section  5.1.2).  While  these 
estimates  represent  our  best  guess  of  the  ultimate  outcome  of  these  cases,  the  estimates  are 
speculative.  Given  that  not  all  receivables  will  be  returned  to  the  Medicare  Trust  Fund,  this 
estimate  probably  overstates  the  ultimate  savings  associated  with  ORT. 

For  all  three  estimates,  ORT  impacts  are  adjusted  based  on  the  double  difference  model,  which  compares 
measures  of  change  (from  the  pre-ORT  period)  are  for  ORT  and  companson  states.  This  is  an  important 
difference  from  the  ORT  Performance  Indicators,  which  report  total  criminal  and  civil  receivables  and 
Medicare  recoveries  for  ORT  cases,  but  do  not  adjust  these  measures  based  on  a  pre/post  or  double 
difference  estimator. 
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Exhibit  5-22:  ORT  Impact  Analyses 

Receivables  Established  from  Criminal,  Civil,  and  CMPL  Cases  and  Administrative  Actions 


ORT  states 

Comparison  states 

date  outcome  observed: 

pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

date  case  opened: 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 

4/Q1 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 
4/95 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 
4/93 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 
4/95 

1  y pe  OT  receivaDic. 

#  of  cases  with  receivable 
established  ( %  ot  total  ) 

10  (10.3%) 

6  (13.3%) 

25  (9.4%) 

17  (25.3%) 

14  (14.1%) 

24  (34.7%.) 

20  (14.3%) 

23  (31.1%) 

Criminal  receivables 

$3,280,331 

$2,017,759^ 

$297,362 

$4,057,472 

$176,204 

$881,335 

$430,771 

$17,031,051 

Civil  receivables 

$5,119,982 

$5,606,664^ 

$63,488,527 

$35,037,747 

$5,578,395 

$10,210,753 

$10,276,793 

$52,117,229 

CMPL  receivables 

$0 

$0 

$2,867,640 

$3,524,000 

$273,000 

$3,662,360 

$918,404 

$1,693,580 

Administrative  receivables 

$586,871 

$0 

$2,4b4,b0b 

5>/,oio,aoy 

<t97  7^7 
9C,  /  ,iOi 

"Cm  Rn'i 

Total  excluding  receivables 
from  administrative  cases  and 
program  savings  (%  change 
from  pre-ORT  period) 

$8,400,313 

$7,624,423^ 

$66,653,529 
(+693%) 

$42,619,219 
(+458%) 

$6,027,599 

$14,754,268 

$11,625,968 
(+93%) 

$70,841,860 
(+380%) 

Total  including  receivables 
from  administrative  cases  and 
program  savings  (%  change 
from  pre-ORT  period) 

$8,987,184 

$7,624,423^ 

$69,118,035 
(+669%) 

$49,937,778 
(+554%) 

$6,055,356 

$15,625,268 

$21,963,571 
(+262%) 

$72,492,670 
(+363%) 

Note:  Outcome  date  is  based  on  date  of  trial/pre-trial  diversion  for  criminal  and  civil  cases.  If  this  date  is  not  recorded  in  CIMS.  then  oulcome  dale  is  based  on  the  date  that  the 


case  closed 
A:  Total  based  on  non-exclusion  cases  only 

B:  Comparison  period  pre-ORT  influenced  totals  have  been  adjusted  to  control  for  differences  in  case  volume  between  ORT  and  comparison  stales  between  4/91  and  3/93 
During  this  period,  197  cases  opened  in  comparison  states  and  97  in  ORT  states,  so  the  reported  figures  represent  2  01  the  actual  amounis  in  CIMS 
Source;  Clf^S   
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Receivables  established  for  cases  opened  during  the  demonstration  period  (i.e.  "pure-ORT" 
cases) 


ORT  was  associated  with  large  changes  in  the  amount  of  receivables  established  from  OI  cases  that  opened 
during  the  demonstration  period  (Exhibit  5-22). 

•  In  ORT  states,  "pure  ORT"  cases  (i.e.,  ORTl  cases,  those  that  opened  dunng  the 
demonstration  period)  resulted  in  receivables  established  of  S66.7  million  (excluding 
program  savings  and  savings  from  cases  that  were  handled  administratively  (i.e..  without 
going  to  a  prosecutor).  This  represented  a  large  increase  from  the  pre-ORT  penod  (as  pre- 
ORT  cases  would  have  been  observed  as  of  the  May  1 995  point  at  which  outcomes  from  the 
pre-ORT  period  are  artificially  censored),  dunng  which  only  S8.4  million  in  receivables  were 
established  (again  excluding  program  and  administrative  receivables). 

•  In  comparison  states.  $11. 6  million  in  receivables  resulted  from  cases  that  opened  dunng  the 
demonstration  period",  an  increase  from  the  pre-ORT  period,  when  S6.0  million  in 
receivables  were  established  (for  "pure  pre-ORT"  cases,  those  that  opened  dunng  the  pre- 
ORT  period). 

Regardless  of  whether  program  savings  are  included  or  excluded  from  the  "receivables  established" 
calculation,  the  estimated  impact  of  ORT  on  receivables  established  is  large.  Excluding  outcomes  from 
administrative  cases  and  program  savings,  ORT  was  associated  with  an  increase  of  more  than 
$54,000,000  in  receivables  established  (taking  into  account  patterns  observed  in  comparison  states  using 
the  double  difference  estimator).  This  represents  a  net  impact  of  more  than  500%.  When  administrative 
cases  and  program  savings  are  also  considered,  the  estimated  impact  of  ORT  on  receivables  established 
is  $36.6  million,  for  an  estimated  ORT  impact  of  more  than  200%. 

We  also  analyzed  the  receivables  established  for  individual  ORT  states: 

Fourteen  of  the  twenty-five  ORT  cases  that  had  identified  receivables  identified  by  the  data 
cutoff  point  resulted  from  cases  developed  by  01  in  Florida,  and  OI  in  Florida  was 
responsible  for  70%  of  the  receivables  established  for  cases  that  opened  during  ORT.  Nearly 
50%  of  the  total  receivables  established  in  ORT  states  for  ORT-period  cases  was  the  result 
of  a  single  civil  case  against  a  Florida  DME  supplier,  which  resulted  in  a  $32.4  million  civil 
judgement.  It  is  difficult  to  know  what  to  make  of  Flonda's  dominance  in  this  category,  but 
it  may  be  partly  due  to  the  large  number  of  fraudulent  DME  suppliers  believed  to  exist  in 
Flonda.  Some  of  the  documented  cases  of  fraud  were  so  clear-cut  that  cases  were  accepted 
for  prosecution  rather  quickly,  and  did  not  require  the  same  amount  of  case  development  work 
as  other  cases. 

•  The  other  four  demonstration  states  had  between  one  (California)  and  four  (Texas)  ORT 
period  cases  that  resulted  in  an  identified  monetary  outcome.  ORT-period  cases  opened  by 


22   There  was  an  additional  $10.3  million  in  administrative  resulting  from  two  cases,  $9.4  million  of 
which  was  the  result  of  a  Wisconsin  home  health  agency  case. 
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01  in  California  resulted  in  only  $294,862  in  receivables,  the  result  of  a  DME  Supplier  case 
opened  in  May  1995.  The  three  Illinois  cases  that  achieved  a  final  outcome  resulted  in  S9.3 
million  in  receivables,  including  a  civil  case  against  a  DME  supplier  that  resulted  in  a  S5.9 
million  settlement. 

•  Nineteen  of  the  ORT  period  cases  that  resulted  in  the  establishment  of  a  receivable  involved 
DME  suppliers.  Four  of  the  cases  involved  nursmg  homes  and  the  other  two  involved  home 
health  agencies. 

Receivables  established  for  cases  that  were  pending  at  the  start  of  the  ORT/pre-ORT  period  (i.e. 
"ORT-influenced"  cases) 

ORT  was  also  associated  with  a  large  increase  in  receivables  established  for  cases  that  were  pending  when 
the  demonstration  started  (Exhibit  5-22). 

•  In  ORT  states.  17  cases  that  were  pending  when  the  demonstration  started  achieved  a 
monetary  outcome  during  ORT.  resulting  in  receivables  established  of  S42.6  million,  and  an 
additional  S7.3  million  in  program  and  administrative  savings.  Nearly  half  of  this  total  was 
the  result  of  a  single  case  (against  a  Texas  DME  supplier),  which  resulted  in  a  S20,8  million 
settlement.  This  was  a  large  increase  relative  to  the  pre-ORT  period.  Six  cases  that  opened 
in  the  pre-pre-ORT  period  (the  two  years  preceding  the  pre-ORT  period)  had  a  monetary 
outcome  during  the  pre-ORT  period,  resulting  in  the  establishment  of  S7.6  million  in 
receivables  (adjusted  for  the  difference  in  the  number  of  pending  cases  observed  at  the  start 
of  the  pre-ORT  period  for  demonstration  and  comparison  states). 

•  In  comparison  states,  cases  that  were  pending  when  the  ORT  period  started  resulted  in 
receivables  or  $70.8  million.  $50.0  million  of  this  total  was  the  result  of  two  cases  against 
Georgia  home  health  agencies.  This  represented  a  large  increase  from  the  pre-ORT  period. 
Twenty-four  cases  that  were  pending  at  the  start  of  the  pre-ORT  period  achieved  a  monetary 
outcomes,  resulting  in  receivables  of  $14.7  million. 

As  discussed  above,  while  the  acmal  opening  of  these  ORT-influenced  cases  cannot  be  attributed  to  ORT, 
(the  cases  were  opened  prior  to  the  start  of  the  demonstration  penod),  ORT  may  have  influenced  the 
development  and  ultimate  outcome  of  these  cases.  To  be  sure,  it  is  not  possible  to  determine  how  the 
outcome  of  these  cases  would  have  been  different  in  the  absence  of  ORT. 

Given  the  pre-post  changes  observed  in  demonstration  and  comparison  states,  ORT  was  associated  with 
a  S5.9  million  increase  in  receivables  resulting  from  cases  that  were  pending  as  of  the  start  of  the 
demonstration  period  (excluding  program  savings  and  receivables  resulting  from  administrative  cases), 
an  increase  of  more  than  80%.  When  program  and  administrative  savings  are  considered,  the  estimated 
ORT  impact  increases  to  $32.3  million.  Note  that  this  impact  is  due  more  to  a  low  volume  of  receivables 
from  pending  cases  in  ORT  states  during  the  pre-ORT  period  ($7.6  million),  relative  to  pending  cases  from 
comparison  states  during  the  pre-ORT  period  ($14.7  million). 
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Conditional  Expected  Receivables  for  Pending  Cases 

Because  there  are  more  than  twice  as  many  cases  pending  in  ORT  states  than  in  comparison  states  (Exhibit 
5-23),  the  measured  impact  of  ORT  on  receivables  established  is  likely  to  widen  as  more  cases  reach  a 
final  outcome.  There  are  also  more  cases  from  the  ORT  period  pending  then  in  the  comparable  (censored) 
pre-ORT  period  (as  it  would  have  been  observed  in  May  1995),  so  estimated  pre-post  differences  are  also 
likely  to  increase  through  time. 

To  estimate  the  expected  receivables  for  pending  cases,  we  used  the  conditional  expected  savings  figures 
reported  in  Exhibit  5-4  and  information  on  the  status  of  pending  cases  at  the  end  of  each  observation 
period  (Exhibit  5-23).  Given  the  extremely  large  standard  deviation  associated  with  the  conditional 
expected  receivables  (the  figures  are  not  statistically  different  from  zero  due  to  the  large  standard 
deviation),  these  estimates  are  not  likely  to  prove  to  be  precise.  But  they  do  represent  our  best  guess,  given 
the  information  currently  available  to  us,  of  what  the  outcome  of  pending  cases  is  likely  to  be. 

The  conditional  expected  receivables  for  the  1 82  ORT  penod/ORT  state  cases  that  were  pending  as  of  the 
cutoff  date  of  our  CIMS  data  is  S85.8  million  (Exhibit  5-23).  This  figure  includes  $66  million  for  pending 
cases  that  had  not  been  accepted  for  prosecution  (167  cases  times  5400,244  per  case).  S14.4  million  for 
the  21  pending  cases  that  had  been  accepted  for  criminal  prosecution  (21  cases  times  5685,857  per  case), 
and  54.6  million  for  the  four  pending  civil  cases  (4  cases  times  51 ,146,356  per  case). 

By  contrast,  the  expected  conditional  receivables  for  ORT  state  cases  that  were  pending  at  the  end  of  the 
pre-ORT  penod  was  $47.4  million  (from  Exhibit  5-22.  these  cases  had  resulted  in  actual  receivables  of 
$42.6  million  as  of  May  1 997). 

The  conditional  expected  receivables  for  the  70  demonstration  period  cases  from  comparison  states  is 
$35.4  million,  more  than  $50  million  less  than  for  ORT  states.  Expected  receivables  were  31%  lower 
dunng  ORT  in  comparison  states  than  in  the  pre-ORT  period,  reflecting  the  lower  volume  of  companson 
state  cases  that  opened  during  ORT. 

Given  the  figures  reported  in  Exhibit  5-23.  we  estimate  that  ORT  will  be  associated  with  an  additional 
$53.1  million  in  receivables  resulting  from  cases  that  are  still  pending.  This  estimate  is  based  on  the 
double  difference  estimator,  and  reflects  the  much  higher  volume  of  non-exclusion  cases  associated  with 
ORT.  This  represents  a  net  increase  in  expected  receivables  of  1 107c,  relative  to  the  pre-ORT  penod.  and 
an  overall  impact  of  162%,  given  the  pre/post  trends  observed  in  comparison  states. 

When  ORT-influenced  cases  are  also  considered  (i.e.,  using  the  broader  0RT2  definition),  the  expected 
impact  associated  with  ORT  decreases  slightly,  but  our  estimate  is  that  ORT  will  still  result  in  an 
additional  $51.8  million  in  receivables  from  pending  cases,  relative  to  what  would  have  occurred  in  the 
absence  of  the  demonstration. 
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Exhibit  5-23:  ORT  Impact  Analyses 


ORT  states 

Comparis 

on  period 

date  outcome  observed: 

pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

date  case  opened: 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 
4/93 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 

4/33 

4/93-3/95 
(pre-ORT) 

4/91-3/93; 
pending 

4/95-3/97 
(pure-ORT) 

4/93-3/95, 
pending 
4/95 

Cases  opened  but 
pending-(No  acceptance  for 
prosecution  or  indictment) 

Number 

99 

24« 

167 

36 

90 

16 

54 

33 

Expected  receivables* 

$39,624,156 

$9,605,856^ 

$66,840,748 

$14,408,784 

$36,021,960 

$6,403,90 
4 

$21,613,176 

$13,208,052 

Accepted  for  criminal 
prosecution  and  pending 

Number 

8 

0 

21 

4 

10 

0 

10 

0 

Expected  receivables* 

$5,486,856 

0 

$14,402,997 

$2,743,428 

$6,858,570 

0 

$6,858,570 

0 

Accepted  for  civil 
prosecution  and  pending 

Number 

2 

0 

4 

0 

7 

0 

6 

0 

Expected  receivables* 

$2,292,712 

0 

$4,585,424 

0 

$8,024,492 

0 

$6,878,136 

0 

Total  expected  savings  for 
pending  cases (%  change 
from  pre-ORT  period) 

$47,403,724 

$9,605,856^ 

$85,829,169 
(+81%) 

$17,152,212 
(+79%) 

$50,905,022 

$6,403,90 
4 

$35,349,882 
(-31%) 

$13,208,052 
(+106%) 

A;  Expected  savings  based  on  conditional  expected  savings  figures  reported  in  Extiibit  5-  3. 

B-  Comparison  period  pre-ORT  influenced  totals  have  been  adjusted  to  control  for  differences  in  case  volume  between  ORT  and  comparison  stales  between  4/91  and  3/93. 

During  this  period,  197  cases  opened  in  comparison  states  and  97  in  ORT  states,  so  the  reported  figures  represent  2  01  the  actual  amounis  in  CIMS 

Source:  CIMS   — — — — — 
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Actual  Monetary  Recoveries  for  ORT  Cases 


HCFA  has  been  tracking  the  funds  actually  recovered  for  ORT  cases.  This  represents  receivables  that 
have  been  collected  by  the  government.  As  of  June  1997.  SI  8  million  had  been  actually  recovered  by 
Medicare  for  cases  identified  by  01  as  ORT-related  (Exhibit  5-24).  S5.5  million  of  this  total  resulted  from 
cases  defined  as  ORT-related  using  the  ORTl  definition  (i.e..  cases  from  ORT  states  with  an  ORT  subject 
type  that  opened  during  the  demonstration  period). 

An  additional  SI  .5  million  in  Medicare  recoveries  is  reported  for  cases  from  ORT  states  that  were  pending 
when  the  demonstration  started.  This  figure  is  likely  to  understate  the  actual  Medicare  recovenes  from 
these  cases,  since  not  all  of  them  cases  are  considered  ORT-related  by  01.  HCFA  did  not  track  recovenes 
for  cases  that  we  defined  as  ORT-related  using  the  0RT2  definition  that  were  not  considered  ORT-related 
by  01. 

Approximately  S4  million  of  the  total  savings  reponed  by  HCFA  could  not  be  linked  to  the  CIMS  data, 
perhaps  due  to  discrepancies  in  the  case  numbers  reported  by  01  and  HCFA.  If  the  unlinked  amounts  are 
assumed  to  have  the  same  distribution  across  slate  and  time  penod  as  records  that  could  be  linked  to 
CIMS.  then  our  estimate  of  actual  Medicare  recovenes  for  ORT  period  cases  increases  to  just  over  S7.0 
million.  This  figure  represents  just  over  10%  of  receivables  established  as  a  result  of  those  cases,  although 
the  percentage  will  certainly  nse  through  time  as  additional  funds  are  recovered  by  Medicare. 


Exhibit  5-24:  ORT  Impact  Analyses 

Medicare  Recoveries  (ORT  states  and  ORT  period  only) 

Period  that  case  opened:         #  of  cases  with  recoveries 

Medicare  Recoveries 

ORT  period  ("Pure  ORT")                               1 2 

$5,464,703 

Pre-ORT  period  ("ORT-  9 

$1,489,288 

influenced") 

Notes:  Data  on  actual  Medicare  recovenes  covers  the  period  through  June  12.  1997 

Sources;  HCFA,  CIMS 

Summary  of  Impact  of  ORT  on  Receivables  Established  and  Monetary  Savings 

Exhibits  5-21  -  5-24  presented  a  range  of  estimated  ORT  impacts,  which  can  be  used  to  establish  upper 
and  lower  bounds  of  the  effects  on  receivables  and  recoveries  associated  with  ORT  (Exhibits  5-25  -  5-28). 
Exhibit  5-28  illustrates  the  alternative  measures  of  the  impact  of  ORT  on  Medicare  recovenes  and 
receivables  identified  that  we  report  in  this  section.  Regardless  of  which  definition  one  used  to  define  ORT 
cases,  and  regardless  of  which  outcomes  are  assumed  for  pending  cases,  the  demonstration  had  a  large 
impact  on  receivables  established  by  OI. 

.  Considering  only  cases  that  opened  during  ORT  ("pure-ORT"  cases),  ORT  was  associated  with 
a  net  impact  of  S50.4  million  in  receivables  identified  for  non-pending  cases  (Exhibit  5-  25. 
Exhibit  5-28).  This  figure  includes  an  estimated  S4.1  in  actual  Medicare  recoveries.  When  the 
outcome  of  pending  cases  is  imputed  using  conditional  expected  receivables,  the  estimated 
impact  for  ORT  period  cases  increases  to  $108  million. 
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Exhibit  5-25:  ORT  Impact  Analyses 

Summary  of  Receivables  Established  and  Medicare  Recoveries  for  Cases  Opened  During  the  Demonstration  Period  (i.e., 
"Pure-ORT"  cases)   — ^— 


ORT-states 

Comparison  states 

Period 

estimated 
recoveries* 

receivables 

iHpntifipH  fnr 

completed 
cases° 

including 
conditional 
expected 
receivables  for 
pending  cases'^ 

estimated 

Medicare 

recoveries* 

receivables 
identified  for 
completed 
cases° 

including 
conditional 
expected 
receivables  for 
pending  cases^ 

Pre-ORT 

$688,741 

$8,400,313 

$55,804,037 

$494,202 

$  6,027,599 

$56,932,621 

ORT 

(Percentage 
change  from  pre- 
ORT  period) 

$5,464,703 

$66,653,529 
^+Dyo  /o) 

$152,482,698 

yr  I  /  O  /o) 

$953,213 
^+93%^ 

$11,625,968 
(+93%) 

$46,975,850 
(-21%) 

Pre-post 
difference 

+$4,775,962 

+$58,253,216 

+$98,678,661 

+$459,010 

+$5,598,369 

-$9,956,771 

Estimated  ORT 
impact" 

+$4,132,446 

+$50,401,878 

+$108,259,832 

Note;  Program  savings  and  receivables  due  to  administrative  cases  excluded. 


A:  Figures  from  Extiibit  5-  24.  Percentage  in  comparison  states  and  the  pre-ORT  period  Is  assumed  to  be  the  same  as  lor  ORT  states  during  the 

demonstration  period. 

B;  Figures  are  from  Exhibit  5-  22. 

C:  Sum  of  figures  on  Exhibit  5-s  22  and  23. 

D;  Double  difference  estimator 


Source:  CIMS 
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Exhibit  5-26:  ORT  Impact  Analyses 

Summary  of  Receivables  Established  and  Estimated  Medicare  Recoveries  for  Cases  Closed  During  the  Demonstration 
Period  (i.e.,  "ORT-influenced"  cases)  — 


ORT-states 

Comparison  states 

Period 

estimated 
actual 
Medicare 
recoveries* 

leceivciuico 
as  reported 
in  CIMS^ 

rckf^oiwahlfkc  inpliiHinn 
1  Ci«dvauiw9  II  iviuuii  1^ 

conditional  expected 
receivables  for 
pending  cases'^ 

estimated 
actual 
Meuicaie 
recoveries* 

receivables 
as  reported 
in  CIMS° 

receivables  including 
conditional  expected 
receivables  for 
pending  cases^ 

Pre-ORT 

$625,126 

$7,624,423^ 

$12,427,351 

$1,209,702 

$14,754,268 

$21,158,172 

ORT 

(Percentage 
change  from  pre- 
ORT  period) 

$3,494,350 
(+458%) 

$42,619,219 
(+458%) 

$59,771,431 
(+381 7o) 

$5,808,324 
(+380%) 

$70,841,860 

^  +  OOU  /O) 

$85,700,722 

1  T  vJ  /Of 

Pre-post 
difference 

+$2,869,224 

+$24,994,796 

+$47,344,080 

+$4,598,622 

+$56,087,592 

+$73,542,550 

Estimated  ORT 
impact" 

+$487,598 

+$5,947,050 

+$9,444,787 

Note;  Program  savings  and  receivables  due  lo  administrative  cases  excluded. 


A:  Figures  from  Exhibit  5-  24.  Percentage  in  comparison  states  and  the  pre-ORT  period  is  assumed  lo  be  the  same  as  for  ORT  stales  during  the  demonstration 
period. 

B;  Figures  are  from  Exhibit  S-  22 

C:  Sum  of  figures  on  Exhibit  5-s  22  and  23. 

D;  Double  difference  estimator 

E:  Volume-adjusted  figures 


Source:  CII^S 
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•  The  estimated  impacts  associated  with  ORT  for  pending  cases  include  S487  thousand  in 
Medicare  recoveries,  $5.4  million  in  receivables  identified,  and  S3. 5  million  for  pending  cases 
(Exhibits  5-25.  5-26.  5-27.  5-28). 

•  Combining  "pure-ORT"  and  "ORT-influenced"  cases,  ORT  was  associated  with  a  net  increase 
of  556.3  million  in  receivables  if  no  outcome  is  assigned  to  pending  cases,  and  an  impact  of 
$1 17.7  million  if  conditional  expected  receivables  are  considered.  The  estimated  impact  on 
Medicare  recovenes  is  $4.6  million  (Exhibits  5-27.  5-28). 

This  amount  is  less  than  the  $141  million  in  Civil  and  Cnminal  receivables  reported  in  the  March  1997 
ORT  Performance  Indicators,  because  it  takes  into  account  what  would  have  happened  anvwav  -  i.e..  it 
is  a  double  difference  estimate  and  therefore  a  more  unambiguously  "incremental"  measure  of  ORT's 
impact.  In  Section  5.5.  we  report  our  esumate  of  the  performance  indicators  used  by  HCFA  and  OIG  to 
measure  the  impact  of  ORT. 


Exhibit  5-27:  ORT  Impact  Analyses 

Summary  of  Impact  of  ORT  on  Medicare  Recoveries  and  Receivables  Established 

Impact  associated  with  ORT  for: 


Period 

estimated 
actual  Medicare 
recoveries* 

receivables 
as  reported 
in  CIMS^ 

receivables  including 
conditional  expected 
receivables  for  pending 
cases'^ 

Pure-ORT  cases  (0RT1) 

$4,132,446 

$50,401,878 

$108,259,832 

ORT-influenced  cases  (0RT2) 

$487,598 

$5,947,050 

$9,444,787 

Total 

$4,620,044 

$56,348,928 

$117,704,619 

Note;  Program  savings  and  receivables  due  to  administrative  cases  excluded. 

A:  Figures  from  Exhibit  5-  24.  Percentage  in  comparison  states  and  the  pre-ORT  period  is  assumed  to  be  the 

same  as  for  ORT  states  during  the  demonstration  period. 

B:  Figures  are  from  Exhibit  5-  22. 

C:  Sum  of  figures  on  Exhibit  5-s  22  and  23. 

D:  Double  difference  estimator 

E:  Volume-adjusted  figures 

Source:  CIMS   
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Exhibit  5:28:  ORT  Impact  Analyses 
Alternative  Measures  of  the  Impact  of  ORT  on  Receivables  Established  and  Medicare  Recoveries 


Cases  opened  in  ORT  period  (i.e.,  pure  ORT  cases) 


Cases  opened  in  pre-ORT  period  and  closed  during  ORT  (i.e.,  ORT  influenced  cases) 


Expected  receivables  from  pending  cases 

$57.8  million 

Receivables  identified  but  not  collected 

$46.3  million 

/Medicare  recoveries^V 

(     $4.1  million  ) 

0RT1  ("Pure-ORT  ■)  impacts 
$4  1  million 


$50  4  million 
$108  million 


Medicare  recoveries  Irom  ORT  period  cases 

Medicare  recoveries  and  receivables  identified 

Medicare  recoveries,  receivables  identilied.  and 
conditional  expected  receivables  lor  pending  cases 


Expected  receivables  from  pending  cases 

$3.5  million 

Receivables  identified  but  not  collected 

$5.4  million 

/(••dicar*  racovarlas^ 

(       $487K  ) 

"ORT  Influenced"  outcomes 
$487  thousand 


$5.9  million 
$9  4  million 


Medicare  recoveries  Irom  ORT  period  cases 

Medicare  recoveries  and  receivables  identilied 

Medicare  recoveries  and  receivables  identified,  and 
conditional  expecled  receivables  for  pending  cases 


Total  ORT  impacts:  Sum  of  "Pure-ORT"  and  "ORT-lnfluenced"  Impacts: 

$4  6  million  Medicare  recoveries  from  ORT  period  cases 

$56  3  million  Medicare  recovenes  and  receivables  identified 

$117  7  million  Medicare  recoveries,  receivables  identified,  and 

conditional  expected  receivables  for  pending  cases 
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5.4 


ORT  Performance  Indicators:  Cases  Explicitly  Identified  as  ORT- 
related  in  CIMS 


A  final  view  of  the  CIMS  data  we  need  to  present  is  the  designation  of  cases  as  ORT  or  non-ORT  by 
CIMS.  using  the  "PK"  code  which  identified  cases  related  to  ORT.  Using  this  definition.  435  non- 
exclusion  cases  and  165  exclusion  cases  have  been  identified  as  ORT-related"'.  We  note  again  thai,  since 
the  "PK"  definition  cannot  be  applied  to  the  pre-ORT  cases  using  the  CIMS  data  we  have,  our  analysis  of 
these  cases  is  a  descnptive  analysis  of  ORT  cases,  not  a  pre/post  or  double  difference  analysis. 

5.4.1     Volume  of  Cases  Identified  as  ORT-related  in  CIMS 
Non-exclusion  Cases 

Of  the  436  non-exclusion  cases  identified  as  ORT-related  in  CIMS.  New  York  and  California  each  had 
more  than  100  cases,  while  Flonda  had  78.  Texas  had  56.  and  Illinois  had  34  (Exhibit  5-29).  Forty-one 
cases  involved  subjects  in  non-ORT  states.  Most  of  the  cases  opened  dunng  the  demonstration  penod, 
although  181  opened  before  ORT  started,  including  48  that  opened  more  than  two  years  before  the 
demonstration  began. 

Overall.  3 1  %  ( 1 38)  of  the  non-exclusion  cases  involved  non-ORT  subject  types  (Exhibit  5-30).  The  most 
common  non-ORT  subject  types  were  medical  practices  (22  cases),  ambulance  companies  (12  cases), 
laboratones  (10  cases),  podiatnsts  (16  cases),  psychologic  practices  ( 1 2  cases),  and  pnvate  citizens  (i.e., 
the  general  public)  (11  cases).  The  "leakage"  of  ORT  efforts  into  this  broad  range  of  subject  types 
suggests  that  the  emphasis  in  the  previous  sections  on  ORT-subject  type  cases  may  result  in  underestimates 
of  the  actual  impacts  of  ORT.  This  leakage  can  be  perfectly  legitimate,  even  under  the  very  structured 
definitions  of  what  constitutes  an  ORT  case,  because  examinations  in  nominally  non-ORT  areas  (such  as, 
for  example,  podiatry)  are  essential  to  see  how  schemes  are  being  perpetuated  in  conjunction  with 
providers  who  are  obviously  ORT-related  (e.g.,  nursing  homes). 

The  individual  ORT  states  varied  in  the  proportion  of  cases  identified  as  ORT-related  in  CIMS  that 
involved  ORT  subject  types.  In  Illinois,  only  12  of  34  cases  involved  an  ORT  subject  type,  compared  to 
83%  of  the  cases  from  New  York  and  79%  of  the  cases  from  Flonda  (Exhibit  5-32). 

Exclusion  Cases 

Nationwide.  165  exclusion  cases  were  identified  in  CIMS  as  ORT-related.  including  88  nursing  home 
cases.  62  DME  supplier  cases,  and  12  exclusions  of  home  health  agencies  (Exhibits  5-31  and  5-32).  Texas 
had  the  most  exclusion  cases,  with  76.  Dlinois,  with  eight  cases,  had  the  fewest  exclusion  cases.  Only  one 
exclusion  case  was  from  a  non-ORT  state. 


23     Overall.  258  of  these  cases  were  opened  dunng  the  ORT  penod  and  1 33  opened  in  the  pre-ORT  period.  Fifty-two  of  the  cases  opened 
pnor  to  the  pre-ORT  penod  (i.e..  before  March  1993). 
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Exhibit  5-29:  Impact  or  ORT  Analysis 
Volume  of  Cases  Identified  as  ORT  Related  by  CIMS  by  states 
Non-exclusion  Cases  Only 
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Source:    CIMS  (Figures  as  of  May  28,  1997) 
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Exhibit  5-30:  Impact  or  ORT  Analysis 
Volume  of  Cases  Identified  as  ORT  Related  by  CIMS  by  states 

Exclusion  Cases  Only 


QNo.  of  Cases 


California 


Florida 


Illinois 


New  York 


Texas 


Source: 
Notes: 


CIMS  (Figures  as  of  May  28.  1997) 

1  Includes  ORT  sub|ect  types  only  (DME  supplier,  Home  Health  Agency.  Nursing  Honne  and  Hospice) 

2.  Comparison  states  include  non-ORT  states  witti  an  Ol  suboHice 

3.  Pre  ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  betwe 


en  4/95-3/97 
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Exhibit  5-31:  Impact  or  ORT  Analysis 
Volume  of  Cases  Identified  as  ORT  Related  by  CIMS  by  Subject  Type 

Non-exclusion  Cases  Only 


ODME 
■  HHA 

□  SNF 

□  Non-CRT 


Source:    CIMS  (Figures  as  of  May  28.  1 997) 

Notes:      1 .  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2  Comparison  states  include  non  ORT  states  witli  an  Ol  subottice 

3  Pre-ORT  period  includes  cases  opened  between  4/93-3/95  Post  ORT  period  includes  cases  opened  between  4/95-3/97 
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Exhibit  5-32:  Impact  or  ORT  Analysis 
Volume  of  Cases  Identified  as  ORT  Related  by  CIMS  by  Subject  Type 

Exclusion  Cases  Only 


Qdme 
■  hha 

□  SNF 


□  Non-CRT 


Source:    CIMS  (Figures  as  of  May  28,  1997) 

Notes:     1  Includes  ORT  subject  types  only  (DME  supplier,  Home  Health  Agency,  Nursing  Home  and  Hospice) 

2.  Comparison  states  include  non  ORT  stales  with  an  Ol  suboffice 

3.  Pre-ORT  period  includes  cases  opened  between  4/93-3/95.  Post  ORT  period  includes  cases  opened  between  4/95  3/97 
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5.4.2    Outcomes  of  Cases  Identified  as  ORT-related  in  CIMS 


Receivables 

Eighty-four  cases  identified  as  ORT-related  in  CIMS  have  resulted  in  the  establishment  of  receivables. 
The  distnbution  of  receivables  by  type  is  shown  in  Exhibit  5-33.  Note  that  our  figures  are  slightly  higher 
than  those  reported  in  the  Operation  Restore  Trust  Performance  Indicators,  since  our  data  cover  two  more 
months  than  the  March  1997  Performance  Indicators.  As  of  May  27.  1997.  these  cases  had  resulted  m  the 
identification  of  $150  million  in  receivables  resulting  from  cnminal.  civil,  and  CMPL  ca.ses.  and  SI 7  9 
million  in  actual  Medicare  recoveries""*.  These  cases  also  resulted  in  S34.0  million  in  administrative 
receivables  and  program  savings,  two  categones  which  are  not  included  in  the  ORT  Performance 
Indicators. 

California  and  Flonda  had  the  largest  amount  of  receivables.  01  in  California  was  responsible  for  nearly 
$49.3  million  of  the  S64.4  million  total  cnminal  receivables,  and  also  had  an  additional  SI  2  million  in  civil 
receivables  (Exhibit  5-34).  01  in  Flonda  produced  S50.7  million  in  civil  receivables  and  the  identification 
of  more  than  S70  million  in  receivables.  Illinois  and  Texas  lagged  far  behind  the  other  ORT  states  in  the 
identification  of  receivables. 

Sanctions 

From  Exhibit  5-35,  224  ORT  cases  remained  pending  (as  of  the  end  of  the  period  covered  by  our  data), 
including  51  that  had  been  accepted  for  criminal  prosecution  and  12  accepted  for  civil  action.  Cases 
identified  as  CIMS-related  in  ORT  resulted  in  54  cnminal  indictments,  40  criminal  trials  (or  pre-tnal 
settlements),  and  44  civil  trials/pre-trial  diversions  (Exhibit  5-  37).  Thirty-three  cases  resulted  in  prison 
sentences,  and  the  178  exclusion  cases  resulted  in  the  exclusion  of  359  individuals  from  the  Medicare 
program  (Exhibit  5-36). 


24    According  to  ihe  ORT  Performance  Indicaiors.  ORT  cases  had  resulted  in  the  identification  of  S67.3  million  in  receivables 
criminal  cases  and  S72.8  in  receivables  from  civil  cases. 
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Exhibit  5-33:  ORT  Performance  Indicators 

Receivables  Established  and  Medicare  Recoveries  for  Cases  Identified  as  ORT-Related  in 
CIMS 


Type  of  receivable  identified/collected            Number  of  cases 

Dollar  amount 

criminal,  civii  ana  umtl  receivauies 

Criminal  receivables 

564,427,066 

wIVM  reCclVaDlcb 

44 

$78,162,098 

CMPL  receivables 

18 

57.480,873 

Total  (excluding  administrative  receivables  and 
program  savings) 

5150.070.037 

Administrative  receivables  and  program  savings 

Administrative  receivables 

9 

$18,200,408 

Program  savings 

28 

ei c  ROD  401 

Total  (including  administrative  receivables  and 
program  savings) 

$184,090,846 

Actual  Medicare  recoveries 

53 

$17,975,660.41 

Totals  are  as  of  May  27,  1997 

Sources:  CIMS,  HCFA. 

Exhibit  5-34:  ORT  Impact  Analyses 

Receivables  Established,  Medicare  Recoveries,  and  Exclusions  by  State  and  Type 

Type  of  receivable 


Number  of  cases 
with  exclusions 
(total  number  of 


State 

Criminal 

Civil 

CMPL 

Administrative 

exclusions) 

California 

549,253,225 

512,585,898 

$3,500,000 

$14,643,855 

22  (38) 

Florida 

56,714,669 

$50,743,615 

$2,729,690 

$80,995 

27  (166) 

Illinois 

$88,160 

5841 ,764 

$397,196 

0 

8(8) 

New  York 

$7,156,898 

54,678,214 

$24,000 

0 

42  (42) 

Texas 

5769,194 

$553,887 

0 

0 

78  (104) 

Totals 

564,427,066 

$78,162,098 

$7,480,873 

$18,200,408 

178  (359) 

Totals  are  as  of  May  27.  1997 
Source:  CIMS.  HCFA   
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Exhibit  5-35:  ORT  Performance  Indicators 

Investigative  Activities  for  Cases  identified  as  ORT-Related  in  CIMS 
Type  of  receivable  identified/collected  Number  of  cases 

Pending  cases:  Total  224 
Pending  cases  accepted  for  criminal  prosecution  51 
Pending  cases  accepted  for  civil  prosecution  12 

Totals  are  as  of  May  27,  1997 
Sources:  CIMS.  HCFA. 


Exhibit  5-36:  ORT  Performance  Indicators 

Sanctions  for  Cases  Identified  as  ORT-Related  in  CIMS 

Type  of  receivable  identified/collected  Number  of  cases 


Sanctions:  Criminal  indictments 

54 

Criminal  triais/pre-tnai  diversions 

40 

Number  of  cases  resulting  in  prison  sentences 

33  (54) 

(number  of  defendants  imprisoned) 

Civil  trials/pre-trial  diversions 

44 

Number  of  cases  with  exclusions  (total  number  of 

178  (359) 

exclusions) 

Totals  are  as  of  May  27,  1997 

Sources:  CIMS 
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5.5  Conclusions 


ORT  had  a  large  impact  on  the  volume  and  sources  of  non-exclusion  cases  opened  by  01.  Across  the  five 
demonstration  states,  volume  increased  by  62%.  relative  to  the  pre-ORT  period.  Given  the  decreased 
volume  observed  in  comparison  states,  the  demonstration  was  associated  with  an  827c  increase  m  the 
number  of  non-exclusion  cases.  While  all  five  ORT  states  had  mcreases  in  volume,  the  largest  increases 
occurred  in  New  York  and  Texas.  ORT  was  associated  with  large  increases  in  the  volume  of  non- 
exclusion  cases  across  all  three  major  ORT  focus  areas.  The  number  of  cases  opened  agamst  home  health 
agencies  and  nursing  homes  more  than  doubled  during  the  demonstration  period.  The  volume  of  exclusion 
cases  decreased  by  21%  during  ORT,  largely  due  to  an  unusually  high  number  of  exclusion  cases 
processed  dunng  the  first  three  months  of  1997.  Especially  important,  we  found  no  evidence  that  the 
increased  volume  resulted  from  the  diversion  of  resources  of  efforts  to  ORT  states  from  companson  states 
within  ORT  regions.  The  positive  outcomes  that  we  have  estimated  as  incremental  ORT  effects-  focusing 
on  cases  from  ORT  states  and  ORT  subject  types-  were  not  obtained  at  the  expense  of  pursuing  cases  in 
non-ORT  states. 

The  ORT  pannership  model  was  associated  with  some  increase  in  the  number  of  non-exclusion  cases 
referred  by  HCFA,  HHS  agencies  other  than  HCFA  (pnmanly  OAS),  HCFA  contractors,  and  state 
agencies.  The  absolute  number  of  calls  that  could  be  traced  to  a  hotline  call  was  small  (13),  but 
represented  a  large  increase  relative  to  either  the  pre-ORT  period  or  to  companson  states.  Across 
demonstration  states,  the  number  of  cases  referred  by  Medicare  beneficiaries,  pnvate  citizens,  and 
employees  of  providers  accused  of  wrongdoing  more  than  doubled  during  ORT,  due  largely  to  higher 
volume  of  referrals  in  New  York  and  Texas.  While  it  is  not  possible  to  establish  a  direct  causal  link 
between  these  referrals  and  ORT  outreach  efforts,  the  increased  referrals  would  be  consistent  with  such 
efforts. 

Estimation  of  the  impact  of  ORT  on  the  identification  of  receivables  and  actual  Medicare  recoveries  was 
complicated  by  the  fact  that  the  majority  of  cases  that  opened  during  ORT  were  pending  as  of  the  data 
cutoff  point.  Obviously,  it  is  not  possible  to  know  now  what  the  ultimate  outcome  of  pending  cases  will 
be.  To  estimate  the  impact  of  ORT  on  case  development  and  outcome  measures,  we  used  artificially 
censored  outcome  measures,  using  the  case  development  dates  recorded  in  CIMS,  allowing  us  to  make 
relatively  unbiased  direct  compansons  of  pre-ORT  and  ORT-period  data.  To  estimate  the  anticipated 
outcome  of  pending  cases,  we  estimated  the  conditional  receivables  of  the  case,  conditional  on  its  point 
in  the  case  development  process.  This  represents  our  best  guess  of  the  ultimate  outcome  for  pending  cases. 
Estimation  of  actual  Medicare  recoveries  (as  opposed  to  receivables)  was  complicated  by  the  lack  of  data 
on  recovenes  for  cases  from  the  pre-ORT  period  or  from  companson  states. 

Regardless  of  which  definition  of  ORT  is  used,  the  demonstration  had  a  large  impact  on  receivables 
established  by  OI,  and,  to  date,  a  necessarily  smaller  impact  on  actual  Medicare  recovenes.  Considenng 
only  cases  that  opened  during  ORT  ("pure-ORT"  cases),  ORT  was  associated  with  an  impact  of  S50.4 
million  in  receivables  identified  for  non-pending  cases.  When  the  outcome  of  pending  cases  is  imputed 
using  conditional  expected  receivables,  the  estimated  ORT  impact  increases  to  SI 08.3  million.  The 
estimated  impact  on  acmal  Medicare  recoveries  was  $4.1  million.  When  ORT-infiuenced  cases  (i.e.,  those 
that  were  pending  when  ORT  started)  are  considered,  the  impact  of  ORT  on  receivables  established 
increases  to  $56.3  million  in  receivables  if  no  outcome  is  assigned  to  pending  cases,  and  an  impact  of 
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$1 17.7  million  if  conditional  expected  receivables  are  considered.  The  estimated  impact  on  Medicare 
recoveries  is  S4.6  million. 

Six-hundred  cases  were  designated  as  ORT-related  in  CCVIS,  including  435  non-exclusion  cases  and  165 
exclusion  cases.  These  are  the  cases  used  to  calculate  the  CIMS-based  ORT  performance  indicators. 
Texas  and  California  each  had  more  than  100  non-exclusion  cases.  Texas  was  also  responsible  for  46% 
of  exclusion  cases. 

Eighty-four  of  these  cases  resulted  in  the  establishment  of  receivables,  including  S64.4  million  from 
criminal  cases,  S78.1  million  from  civil  cases,  and  $7.5  million  from  CMPL  cases.  California  and  Fionda 
were  each  responsible  for  more  than  $50  million  in  receivables,  while  Illinois  and  Texas  lagged  far  behind 
the  other  ORT  states.  By  May  27,  1997,  these  cases  had  resulted  in  $17.9  million  in  actual  Medicare 
recoveries.  In  addition,  these  cases  resulted  in  the  exclusion  of  359  individuals  from  the  Medicare 
program  and  the  imprisonment  of  54  defendants.  Because  more  than  200  cases  identified  as  ORT-related 
were  still  pending  (as  of  the  point  covered  by  our  data),  these  figures  are  likely  to  increase  through  time. 
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Appendix  6 

Analysis  of  Data  from  the  OAS  Audit  Information 
Management  System  (AIMS) 

6.0  Introduction 

Office  of  Audit  Service  (OAS)  audits  are  intended  to  identify  investigative  targets  and  questioned  charges, 
and  to  make  recommendations  for  policy  and  program  changes  to  make  Medicare  and  Medicaid  less 
vulnerable  to  fraud  and  abuse.  ORT  has  been  associated  with  changes  in  both  the  volume  and 
characteristics  of  audits  being  performed  by  OAS.  Due  in  large  part  to  ORT  travel  funds,  OAS  has  been 
able  to  pursue  a  much  larger  number  of  provider-specific  audits  than  in  the  past.  Many  of  these  audits  are 
intended  to  uncover  cases  of  fraud,  either  to  assist  an  ongoing  01  investigation  or  lead  to  referrals  to  01. 
ORT  has  also  been  associated  with  greater  collaboration  between  OAS  and  HCFA,  especially  in  reviews 
of  home  health  agencies  (see  Case  Studies  of  the  ORT  regions  in  Appendix  1 ). 

ORT  had  a  very  large  impact  on  the  volume  of  audits  of  ORT  focus  area  providers  performed  by  OAS. 
Between  1990  and  1994,  OAS  opened  17  audits  of  ORT  focus  area  providers  located  in  one  of  the  ORT 
states  (this  figure  excludes  canceled  audits).  Between  January  1995  and  April  1997  (the  end  of  the  period 
covered  by  our  data),  1 16  audits  of  ORT  focus-area  providers  opened  across  the  five  demonstration 
states.  In  comparison  states,  there  were  30  ORT-type  audits  opened  in  both  the  pre-  and  post-ORT 
periods.  Using  a  somewhat  different  definition,  133  audits  are  identified  as  ORT-related  in  the  OAS 
Audit  Information  Management  System  (AIMS). 

Nationwide,  of  the  21,686  staff  days  devoted  to  ORT  focus-area  audits  by  OAS  between  1995-1997, 
18,920  (87%)  have  been  related  to  ORT-type  audits  from  ORT  states,  compared  to  2,766  for  ORT-type 
audits  in  non-ORT  states.  During  ORT,  across  ORT  states,  auditors  have  devoted  more  than  five  times 
as  many  staff  days  to  audits  of  ORT-type  providers  than  they  did  on  average  between  1990-1994. 

Although  ORT  travel  funds  proved  invaluable  in  allowing  OAS  to  pursue  the  type  of  provider-specific 
audits  that  had  not  been  pursued  in  the  past,  the  agency  was  able  to  add  only  a  few  additional  staff  with 
ORT  funds.  Given  the  limited  additional  staff  made  possible  by  ORT,  some  diversion  of  resources  (either 
from  within  ORT  states  or  from  other  states)  from  other  types  of  audits  was  necessary  in  order  to  achieve 
this  increase  in  volume.  The  demonstration  was  associated  with  a  70%  increase  in  overall  staff  days 
devoted  to  audits  of  all  types,  suggesting  that  auditors  from  non-ORT  states  may  have  diverted  some  of 
their  efforts  to  focus  on  audits  within  ORT  states. 

Data  in  the  OAS  audit  tracking  system  allow  us  to  measure  changes  in  collaboration  between  OAS  and 
other  ORT  partners  associated  with  ORT.  This  information  provides  a  complement  to  what  we  have 
learned  from  our  discussions  with  the  ORT  regions.  The  number  of  audits  described  as  related  to  an  OI 
investigation  increased  from  3  between  1990-1994  to  15  during  ORT,  indicating  greater  collaboration 
between  OAS  and  01.  From  our  site  visit  discussions,  it  is  clear  that  ORT  has  led  to  increased 
collaboration  between  OAS  and  HCFA.  All  16  audits  identified  as  involving  collaboration  between  OAS 


Abt  Associates  Inc. 


Appendix  6:  Analysis  of  AIMS 


6-1 


and  HCFA  have  occurred  as  a  result  of  ORT,  as  such  collaboration  did  not  occur  prior  to  the 
demonstration. 

Many  audits  that  have  identified  questioned  charges  have  resulted  in  referrals  to  01  for  potential  criminal 
prosecution.  According  to  the  March  1 997  ORT  Performance  Indicators,  these  audits  could  ultimately 
result  in  questioned  costs  of  $81  million.  Because  audits  with  referrals  to  OI  have  been  effectively  placed 
on  hold  pending  the  outcome  of  criminal  investigations,  no  Final  Report  has  been  issued,  and  the  potential 
disallowances  identified  from  these  audits  are  not  yet  reported  in  AIMS,  (questioned  charges  are  not 
reported  in  AIMS  until  after  the  Final  Report  is  issued).  Through  April  1997,  nine  ORT  audits  identified 
$21 .6  million  in  questioned  costs.  Our  data  do  not  allow  us  to  track  how  much  of  this  amount  was  actually 
recovered  by  the  government.  One  other  audit  (of  a  Florida  home  health  agency),  which  opened  in  1 994 
but  was  still  pending  when  ORT  began,  and  is  identified  as  ORT-related  in  AIMS,  resulted  in  $25.9 
million  in  questioned  costs. 

This  section  presents  our  estimates  of  changes  in  audit  volume  and  characteristics  associated  with  ORT. 
These  estimates  are  based  on  comparisons  of  ORT  focus-area  audits  opened  in  ORT  states  during  the 
demonstration  period  to  those  from  the  pre-demonstration  period  and  from  a  group  of  comparison  states. 

6.1    Data  Source  and  Evaluation  Strategy 

6.1.1  Data  source 

OAS  audits  are  tracked  by  the  Audit  Information  Management  System.  It  contains  records  from  all  OAS 
audits.  AIMS  reports  audit  referral  sources,  subject  types,  subject  location,  and  information  on  the 
development  of  audits  through  the  audit  flow  cycle. 

The  AIMS  extract  that  we  received  contains  information  on  more  than  41,000  audits.  Of  the  audits  that 
opened  after  1990,  only  231  either  involved  an  ORT  focus-area  provider'  or  were  identified  by  OAS  as 
related  to  ORT." 

6.1.2  Evaluation  strategy 

Our  basic  approach  for  estimating  the  impact  of  ORT  on  OAS  audits  is  to  compare  measures  of  change 
(from  the  pre-ORT  to  the  ORT  period)  for  ORT  states  and  a  group  of  comparison  states.  The  assumption 
undedying  this  approach  is  that  any  change  in  the  year-to-year  time  trend  that  might  have  occurred  around 
the  time  that  the  ORT  demonstration  started  would  have  had  the  same  effects  in  both  ORT  and  comparison 
states.  This  methodology  offers  greater  protection  against  bias  from  unrelated  factors  that  might  have 
caused  changes  in  audit  volume  and  outcome  in  ORT  states  even  in  the  absence  of  the  ORT  intervention. 


1  AIMS  directly  identifier  audits  involving  nursing  homes  or  home  health  agencies,  but  not  audits  of  DME  suppliers  or  hospices.  Audits 
of  DME  suppliers  and  hospices  have  to  be  inferred  based  on  information  in  the  audit  description  field.   DME  audits  were  identified 
based  on  the  presence  of  "DME",  "Supplier."  "Dressing"  (i.e..  for  surgical  dressing  suppliers),  "body"  (i.e.,  for  body  jackets) ,  or 
"DMEPOS"  in  the  audit  description  field    Hospices  were  identified  based  on  the  presence  of  "hospice"  in  the  audit  description  field 

2  ORT-related  audits  are  identified  based  on  the  presence  of  "ORT'  in  the  audit  descnption  field.  AIMS  identified  1 35  audits  as  ORT- 
related    12.3  of  these  involved  ORT  auditee  types,  and  there  were  also  3  hospital  audits.  2  audits  of  physicians,  and  7  audits  of  other 
medical  professionals  identified  as  ORT-related. 
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/.        Defining  ORT  Audits 


We  define  ORT  audits  based  on  the  auditee  type,  auditee  state,  and  the  year  that  the  audit  opened.  Audits 
of  ORT  focus-area  providers  (including  hospices)  from  ORT  states  that  opened  in  1995  or  1996  are 
considered  to  be  ORT  audits.  Because  the  set  of  audits  identified  as  ORT-related  using  this  method  and 
those  identified  as  ORT-related  based  on  information  in  the  AIMS  audit  description  field,  we  do  not 
present  separate  estimates  of  the  impacts  of  ORT  based  on  the  identifier  contained  in  AIMS\ 

2.  Defining  Comparison  States 

As  we  did  for  our  analyses  of  CIMS  contained  in  Appendix  5,  we  included  as  comparison  states  all  non- 
ORT  states  with  an  OI  suboffice.  The  complete  list  of  comparison  states  can  be  found  in  Exhibit  5-4. 
Because  of  its  status  as  a  quasi-ORT  state,  we  excluded  audits  of  Louisiana  providers  from  our  comparison 
group. 

To  investigate  whether  auditors  from  non-ORT  states  diverted  their  efforts  to  audits  within  the  ORT  states, 
two  additional  comparison  state  specifications  were  used: 

•  ORT  states  vs.  non-ORT  states  within  ORT  regions^ 

•  ORT  regions  vs.  non-ORT  regions 

Measures  of  this  sort  do  not  exactly  track  the  potential  "opportunity  costs"  of  ORT,  but  they  do  help  to 
give  us  a  rough  idea  of  whether  increases  in  ORT-type  audits  are  associated  with  notable  corresponding 
decreases  in  other  work.  The  first  comparison  allows  measurement  of  whether  efforts  in  non-ORT  states 
were  diverted  in  order  to  focus  on  audits  within  ORT  states.  If  for  example,  audit  volume  increased  in 
an  ORT  state  but  decreased  by  a  similar  magnitude  across  other  states  in  its  region,  it  seems  reasonable 
to  infer  that  much  of  the  impact  observed  in  the  ORT  states  resulted  from  diversion  of  efforts  from  other 
states  in  the  region.  If  substitution  of  resources  is  confined  within  the  region,  the  second  comparison 
allows  for  a  measure  of  the  impact  of  ORT  that  is  not  distorted  by  this  type  of  diversion  of  resources.' 

3.  Defining  the  ORT  and pre-ORT  Periods 

The  date  fields  included  in  the  AIMS  data  we  received  permit  us  to  identify  the  year  that  audits  opened 
and  the  date  of  the  most  recent  entry  for  the  record  into  the  AIMS  system. We  included  audits  that 
opened  between  1990-1994  in  the  pre-ORT  period.'  Audits  that  opened  in  1995  that  closed  prior  to  the 


This  practice  contrasts  with  the  approach  we  followed  in  our  analysis  of  CIMS.  See  Appendix  5  of  this  repon 

4  We  define  region  based  on  the  states  served  by  the  HCFA  Regional  Office, 

5  Note  that,  with  respect  to  cases,  we  did  not  find  evidence  of  either  diversion  or  spillover  effects,  likely  because  of  the  staff  increases 
associated  with  ORT  (see  Appendix  .S). 

6  Note  that,  unlike  CIMS.  our  AIMS  data  did  not  identify  the  month  that  audits  were  opened.  We  designated  all  1995  audits  as  ORT 
audits,  a  somewhat  ovennclusive  specification, 

7  Note  that  this  five  year  penod  is  longer  than  the  26-month  base  period  used  in  our  analyses  of  cases  (Appendix  5),  The  longer  penod 
was  necessary  to  get  a  more  reliable  pre-ORT  baseline. 
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April  1,  1995  ORT  starting  date  were  also  counted  as  pre-ORT  audits.*  Audits  from  1995  (that  had 
activity  after  April  1 ,  1 995),  1 996.  and  1 997  were  counted  as  ORT  period  audits.  Because  audits  tend  to 
be  completed  more  quickly  than  cases  that  result  in  prosecution,  and  because  the  analyses  contained  in  this 
appendix  emphasize  measures,  such  as  audit  volume,  auditee  type,  and  the  focus  and  output  of  audits,  that 
are  recorded  in  AIMS  prior  to  the  issuing  of  the  Final  Report,  none  of  the  elaborate  artificial  censoring 
methods  used  in  the  analysis  of  CIMS  data  were  required.  In  view  of  the  extremely  low  pre-ORT  volume 
of  ORT  focus-area  audits,  any  overcount  is  likely  to  be  trivial. 

6.2    Estimates  of  ORT  Impacts 

Our  discussion  of  the  impacts  of  ORT  on  OAS  audits  is  divided  into  four  sections.  First,  we  evaluate 
changes  in  audit  volume,  subject  types,  and  requesting  agencies  associated  with  ORT.  Second,  we 
evaluate  how  ORT  has  affected  the  characteristics  of  audits  being  performed  by  OAS.  Third,  we  discuss 
the  impact  of  ORT  on  the  identification  of  questioned  costs  by  OAS.  Fourth,  we  attempt  to  make  rough 
estimates  of  some  of  the  "opportunity  costs"  of  ORT  in  terms  of  reduced  volume  of  non-ORT  audits. 

6.2.1     Impact  of  ORT  on  Audit  Volume,  Subject  Types  and  Requesting  Agencies 
Audit  Volume 

ORT  was  associated  with  an  extremely  large  increase  in  the  volume  of  audits  of  ORT  subject  type 
providers  (Exhibit  6-1 ).  During  ORT,  1 16  ORT-type  audits  opened  across  the  five  demonstration  states, 
compared  to  only  1 7  ORT-type  audits  that  opened  between  1 990- 1 994  ORT.'  Across  comparison  states, 
there  were  30  audits  each  in  the  pre-ORT  and  ORT  periods. 

Another  way  to  examine  changes  in  volume  associated  with  ORT  is  to  analyze  OAS  staff  time  spent  on 
ORT-type  audits.  ORT  clearly  had  a  major  impact  on  the  priorities  of  OAS  auditors.  During  ORT,  18,920 
staff  days  were  devoted  to  ORT  focus-area  audits  in  demonstration  states,  compared  to  3,651  staff  days 
in  the  pre-ORT  period,  an  increase  of  4187f .  In  comparison  states,  5.693  staff  days  were  devoted  to  ORT 
focus-area  audits  in  the  pre-ORT  penod.  compared  to  2,766  during  ORT.  a  decrease  of  5 1  %. 

Volume  by  State 

The  increase  in  audit  volume  associated  with  ORT  was  spread  across  all  five  demonstration  states  (Exhibit 
6-2).  Only  Florida  had  more  than  five  audits  of  ORT-focus  area  providers  during  the  pre-ORT  period, 
and  two  states,  Illinois  and  New  York,  had  no  audits  of  ORT-focus  area  providers  initiated  between  1990 
and  1994.  During  ORT,  Florida  had  the  most  ORT  audits,  but  ORT  was  a.ssociated  with  a  large  increase 
in  audit  volume  for  four  of  the  five  ORT  states.  Illinois,  which  opened  only  eight  audits  of  ORT  subject- 
type  providers,  was  the  only  state  where  ORT  was  not  associated  with  a  large  increase  in  the  volume  of 
audits  opened. 


8  Our  AIMS  exiraci  did  nol  repon  the  actual  dale  that  an  audit  closed  Closing  date  was  inferred  based  on  the  complelion/cancellation 
code  and  the  date  of  the  most  recent  AIMS  entr>'  related  to  the  audit 

9  Note  that  for  this  repon.  we  excluded  audit  records  that  appeared  to  have  been  established  in  AIMS  merely  for  the  purpose  of  recording 
ORT  expenditures  (e  g  .  audit  records  related  to  attending  conferences  and  meetings).  We  also  excluded  audits  that  were  canceled  pnor 
to  completion  or  that  had  zero  cumulative  staff  days 
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Exhibit  6-1 :  ORT  Impact  Analyses 
Volume  of  Audits  Opened 


nPre-ORT 
■  Post-ORT 


Non-ORT  Slales  ORT  Slates 


Source:  AIMS  (Data  through  April  1997) 
Notes: 

1 .  Includes  ORT  auditee  types  only  (DME  supplier,  home  health  agency,  nursing  home,  hospice). 

2.  Non-ORT  states  include  all  non-ORT  states  with  an  Ol  suboffice. 

3.  Pre-ORT  period  includes  audits  opened  1990-94;  ORT  period  includes  audits  opened  in  1995-97 

4.  Excludes  cancelled  audits. 
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Exhibit  6-2:  ORT  Impact  Analyses: 
Audit  Volume  by  State 
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Source:  AIMS  (Data  through  April  1997) 
Notes: 

1 .  Includes  ORT  auditee  types  only  (DME  supplier,  home  health  agency,  nursing  home,  hospice). 

2.  Non-ORT  states  include  all  non-ORT  states  with  an  01  suboffice. 

3.  Pre-ORT  period  includes  audits  opened  1990-94;  ORT  period  includes  audits  opened  in  1995-97. 

4.  Excludes  cancelled  audits. 
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In  terms  of  staff  days,  OAS  in  Texas  devoted  the  most  resources  to  ORT  (Exhibit  6-3).  Auditors  there 
devoted  5,892  staff  days  to  ORT,  compared  to  757  staff  days  on  audits  of  ORT  focus-area  providers 
during  the  pre-ORT  period.  Across  all  five  states,  ORT  was  associated  with  a  substantial  change  in  how 
OAS  auditors  spent  their  time.  Even  in  Illinois,  which  opened  relatively  few  audits  during  ORT.  2.374 
staff  days  were  devoted  to  audits  of  ORT-type  providers.  This  total  was  not  much  less  than  that  of  OAS 
in  New  York,  which,  however,  opened  three  times  as  many  ORT-related  audits  as  Illinois,  using  that 
comparable  volume  of  staff  days.  The  reason  for  comparable  staff  inputs  but  different  actual  number  of 
audits  opened  may  be  due  to  a  change  in  audit  practices  in  New  York.  During  ORT,  OAS  in  New  York 
emphasized  "short  form"  audits  to  identify  and  document  suspicious  behavior  quickly  (see  Appendix  1 ). 

Subject  Types 

Thirteen  of  the  17  pre-ORT  audits  from  demonstration  states  involved  home  health  agencies.  During 
ORT,  there  were  large  increases  in  volume  for  all  four  major  ORT  focus-areas  (Exhibit  6-4).  Due  in  large 
part  to  the  reviews  of  SNFs  performed  in  most  ORT  states,  the  number  of  nursing  home  audits  increased 
from  two  to  45  during  ORT.  The  number  of  home  health  agency  audits  increased  from  13  to  41 .  OAS 
opened  no  audits  of  hospices  between  1990  and  1994,  but  the  additional  scrutiny  placed  on  hospices 
(especially  those  with  aberrant  numbers  of  long-term  patients  or  patients  with  non-terminal  conditions) 
resulted  in  the  opening  of  1 1  audits  of  hospices.  The  number  of  audits  of  DME  suppliers  increased  from 
two  to  19,  with  eight  of  these  audits  conducted  by  OAS  in  New  York. 

Requesting  Agency 

During  ORT,  OAS  appeared  to  be  more  proactive  in  initiating  audits.  Across  the  five  demonstration 
states,  75  ORT  period  audits  (64%  of  the  total)  were  self-initiated  (i.e.,  OAS  was  identified  in  AIMS  as 
the  requesting  agency)  (Exhibit  6-5).  In  the  pre-ORT  period,  the  demonstration  states  opened  just  10  self- 
initiated  audits.  About  50%  of  audits  in  comparison  states  in  both  the  pre-  and  post-ORT  periods  were 
self-initiated.  These  data  may  reflect  the  influence  of  an  explicit  ORT  goal:  to  make  selection  of  audit 
targets  more  data  driven  and  the  follow-up  on  findings  of  impropriety  by  looking  for  other  similar 
instances.  Both  of  these  goals,  if  achieved,  would  yield  more  self-referrals. 

ORT  partners  were  the  requesting  agency  for  most  audits  that  were  not  self-initiated  by  OAS.  In  ORT 
states,  OI  requested  nine  audits,  compared  to  two  in  the  pre-ORT  period.  OI  requested  audits  also 
increased  in  comparison  states,  from  four  to  eight.  During  ORT,  OP  DIV  (which  is  most  often  HCFA 
for  audits  of  ORT  subject  types)  was  listed  as  the  requesting  agency  for  19  demonstration  state  audits.  OP 
DIV  was  the  requesting  agency  for  only  three  audits  in  the  pre-ORT  period  (all  three  were  from  ORT 
states).  The  increase  was  due  mainly  to  Florida's  reviews  of  SNFs.  17  of  which  listed  OP  DIV  (i.e., 
HCFA)  as  the  requesting  agency.  This  enormous  increase  in  collaboration  referrals  is  also  a  concise 
reflection  of  what  ORT  sought  to  achieve,  although  in  this  case  the  effects  were  more  or  less  confined  to 
a  single  ORT  state. 
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Exhibit  6-3:  ORT  Impact  Analyses 
Cumulative  Staff  Days 
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Source:  AIMS  (Data  through  April  1997) 
Notes: 

1 .  Includes  ORT  audltee  types  only  (DME  supplier,  home  health  agency,  nursing  home,  hospice). 

2.  Non-ORT  states  include  all  non-ORT  states  with  an  01  subotfice. 

3.  Pre-ORT  period  includes  audits  opened  1990-94;  ORT  period  includes  audits  opened  in  1995-97. 

4.  Excludes  cancelled  audits. 
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Exhibit  6-4:  ORT  Impact  Analyses 
Audit  Volume  By  Auditee  Type 


Auditee  Type 

ORT  states 

Comparison  states 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

DME  supplier 

2 

19 

2 

3 

Home  health  agency 

13 

41 

21 

17 

Hospice 

0 

11 

0 

0 

SNF 

2 

45 

7 

10 

Totals 

17 

116 

30 

30 

Note:  Canceled  audits,  audits  witti  zero  cumulative  staff  days,  and  audit  records  established  for  accounting  purposes  are  excluded. 
Includes  audits  of  ORT  focus-area  providers  only. 

Source:  AIMS   


Exhibit  6-5:  ORT  Impact  Analyses 

Impact  of  ORT  on  audit  characteristics:  Requesting  Agency 


Requesting  Agency 

ORT  states 

Comparison  state 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

OAS 

10  (59%) 

75  (66%) 

14  (47%) 

16  (53%) 

OCFAA 

0  (0%) 

1  (1%) 

0  (0%) 

1  (3%) 

IG 

0  (0%) 

3  (3%) 

3  (10%) 

1  (3%) 

01 

2  (12%) 

9  (8%) 

4  (13%) 

8  (27%) 

RIGA 

1  (6%) 

1  (1%) 

6  (20%) 

1  (3%) 

PRO 

0  (0%) 

0  (0%) 

2  (7%) 

0  (0%) 

OP  DIV* 

3  (18%) 

19  (16%) 

0  (0%) 

1  (3%) 

Assist  USAO 

0  (0%) 

2  (2%) 

0  (0%) 

1  (3%) 

Other  Federal  Dept 

0  (0%) 

5  (4%) 

0  (0%) 

0  (0%) 

Other 

1  (6%) 

1  (1%) 

1  (3%) 

1  (3%) 

Totals 

17 

116 

30 

30 

Note:  Canceled  audits,  audits  with  zero  cumulative  staff  days,  and  audit  records  established  for  accounting  purposes  are 
excluded.  Includes  audits  of  ORT  focus-area  providers  only. 

A-  Includes  HCFA,  Public  Health  Services,  Administration  for  Children  and  Families,  Office  of  the  Secretary,  Administration  on 
Aging,  Agency  for  Toxic  Substances  and  Disease  Registry,  and  Agency  for  Health  Care  Policy  and  Research. 

Source:  AIMS   
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6.2.2    Characteristics  of  Audits  Being  Pursued  as  a  Result  of  ORT 

In  addition  to  its  impact  on  audit  volume,  ORT  appeared  to  be  associated  with  changes  in  the 
characteristics  of  audits.  We  focus  on  changes  associated  with  ORT  in  the  audit  approach,  as  measured 
by  the  total  staff  days  required  to  complete  audits,  the  output  associated  with  audits,  and  the  focus  area  of 
work  being  done  for  ORT  audits. 

Staff  Days  Required  to  Complete  Audits 

On  average,  in  terms  of  staff  inputs,  ORT-type  audits  were  less  complex  during  ORT  than  in  the  pre-ORT 
period.  Analysis  of  recorded  staff  days  allows  us  to  measure  whether  ORT  was  associated  with  changes 
in  the  complexity  of  audits,  at  least  as  measured  by  the  intensity  of  staff  resources. 

Seventy  ORT-period  audits  from  demonstration  states  are  identified  as  "complete"  in  AIMS,  based  on  the 
completion/cancellation  code  which  indicates  the  current  status  of  audits.  These  audits  required  an 
average  of  122  staff  days.  The  17  pre-ORT  audits  from  demonstration  states  took,  on  average,  190  staff 
days  each.  There  are  still  46  ORT-period  audits  pending,  which  may  increase  the  average  number  of  staff 
days  devoted  to  ORT-period  audits,  but  these  findings  are  consistent  with  what  we  learned  from  OAS 
auditors  in  the  field,  who  reported  that  ORT  audits  tended  to  emphasize  the  identification  of  aberrant 
providers  using  a  smaller  number  of  claims-  there  was  a  greater  emphasis  on  generating  referrals  for  OI 
or  assisting  01  in  ongoing  criminal  investigations.  In  addition,  there  were  deliberate  efforts  (e.g.,  in 
California  and  New  York)  to  develop  "faster  turnaround"  methods  for  identifying  fraud  and  referring 
targets  to  law  enforcement  agencies. 

Work  Output 

Another  measure  of  whether  or  not  OAS's  practices  have  changed  is  the  output  of  audits.  AIMS  includes 
information  on  the  output  produced  by  each  audit.  Consistent  with  the  ORT  Partnership  model,  there  was 
a  large  increase  in  the  number  of  audits  that  involved  audit-assist  work.  These  audits  do  not  result  in  the 
issuing  of  a  formal  audit  report,  but  rather  a  report  that  is  issued  internally  for  OTs  use  in  developing  a 
case. 

During  ORT,  36  audits  (31%)  in  ORT  states  had  a  work  output  code  related  to  assist-audit  work,  compared 
to  3  audits  (18%)  in  the  pre-ORT  period  (Exhibit  6-6).  These  were  primarily  audits  where  OAS  assisted 
either  01  with  a  criminal  investigation  or  HCFA  as  part  of  the  surveys  of  SNFs. 

ORT  has  also  been  associated  with  a  large  increase  in  the  volume  of  audits  related  to  grantee  or  contractor 
operations.  Many  of  these  audits  are  provider-specific  audits  of  cost  reports  or  claims.  These  audits 
include  the  reviews  of  hospice  eligibility  and  many  of  the  ORT  reviews  of  home  health  agencies. 
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Exhibit  6-6:  ORT  Impact  Analyses 

Impact  of  ORT  on  Audit  Characteristics:  Work  Output 


ORT  states 

Comparison  state 

worK  uuipui 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

Assist  audit  work 

3  (18%) 

36  (31%,) 

8  (27%) 

7  (23%) 

Audit  report  on  departmental 
operations 

0  (0%) 

2  (2%) 

0  (0%) 

0  (0%) 

Audit  report  on 
grantee/contractor  operations 

5  (29%) 

62  (53%) 

14  (47%) 

12  (40%) 

Survey  work* 

2  (12%) 

4  (3%) 

3(10%) 

4(13%) 

Nationwide  reporting 

4  (24%) 

2  (2%) 

0  (0%) 

0  (0%) 

Staff  analysis  :  No  report 

1  (6%) 

2  (2%) 

1  (3%) 

5(17%) 

Other^ 

2(12%) 

8  (7%) 

4  (13%) 

2  (7%) 

Totals 

17 

116 

30 

30 

A;  Survey  refers  to  phase  three  of  the  audit  process.  All  audits  should  be  initially  coded  with  a  work  output  code  indicating  that 
a  report  will  be  issued,  but  if  an  audit  is  ended  after  the  survey  phase,  the  work  output  code  should  be  changed  to  "Survey 
work." 

B;  Includes  audits  with  the  following  work  ouput  codes:  audit  work  prepared  by  CPA  under  contract.  Follow-up  Audit  Report. 
Staff  Support,  Preaward  cost  proposal  audit  award,  and  Cost  Allocation. 

Source:  AltvIS 


Purpose  of  Audits 

A  final  tracer  of  possible  changes  in  OAS  practices  concerns  the  focus  of  audits.  Under  ORT.  the  plan 
was  for  OAS  to  focus  more  on  misbehavior  by  specific  providers,  rather  than  on  identifying  program 
vulnerabilities  that  encourage  provider  abuse.  The  difference  here  is  more  than  semantic.  For  example, 
for  provider-specific  audits,  OAS  would  draw  a  random  sample  of  claims  for  a  specific  suspicious 
supplier,  while  program  audits  typically  involve  random  samples  of  a  cross-section  of  providers.  In  the 
latter  case,  any  given  provider  typically  would  have  at  most  only  a  few  claims  in  the  sample,  thus  creating 
technical  obstacles  to  identifying  illegal  behavior  by  specific  providers. 

AIMS  contains  a  field  which  describes  the  purpose  of  each  audit.  ORT  was  associated  with  a  large 
increase  in  the  number  of  audits  devoted  to  issues  of  program  compliance,  services  provided/delivery 
systems,  and  alleged  criminal  activities  (Exhibit  6-7).  Audits  related  to  program  compliance  include  many 
of  the  provider-specific  audits,  including  those  that  involve  assist-audit  work  for  OI.  Due  in  large  part  to 
Florida's  reviews  of  SNFs,  the  number  of  audits  that  focused  on  services  provided/delivered  increased 
from  zero  to  29  during  ORT.  These  reviews  sought  to  verify  whether  billed  services  were  ordered, 
medically  necessary,  provided  and  appropriately  performed. 
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Exhibit  6-7:  ORT  Impact  Analyses 

Impact  of  ORT  on  Audit  Characteristics:  Audit  Focus 


ORT  states 

Comparison  state 

Audit  Focus 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

Program  compliance 

10  (58%) 

59  (51%) 

13  (43%) 

12  (40%) 

Program  management 

1  yo  /o) 

O  (00/  \ 

1  (•J /o) 

<L\I  /o) 

Arlmini<?trati\/p  rost"; 

i\  V-J  1  1  III  IIOll  CtllV        w  W  O  \  <J 

1  (b  /o) 

J  (j  /o) 

4  (\J  /o) 

O  17°/  \ 

.^prvirpQ  nrn\/iHpH/Hpli\/pr\/ 

systems 

0  (0%) 

29  (25%) 

0  (0%) 

1  (3%) 

Alleged  criminal  activities 

2  (12%) 

9  (8%) 

8  (27%) 

10  (33%) 

Non-Federal  audit  report 
activities 

0  (0%) 

5  (4%) 

2  (7%) 

1  (3%) 

Grant/contract  activities 

2  (12%.) 

2  (2%) 

2  (7%) 

0  (0%) 

Other^ 

1  (6%) 

6  (5%) 

0  (0%.) 

2  (7%) 

Totals 

17 

116 

30 

30 

A;  Includes  personnel  activities,  direct  costs,  data  processing  and  security  activities,  and  not  applicable. 
Source:  AIMS 


Collaboration  Between  OAS  and  OI 

Our  site  visit  discussions  suggested  that  there  had  been  increased  collaboration  between  OAS  and  01 
during  ORT.  It  is  possible  to  identify  audits  where  OAS  assisted  with  an  01  investigation  using 
information  contained  in  the  AIMS  audit  description  field.'"  Across  the  ORT  states,  15  audits  were 
opened  to  assist  01,  compared  to  only  three  in  the  pre-ORT  period  (Exhibit  6-8).  Each  ORT  state  had  at 
least  two  audits  that  were  01  assists.  New  York  had  five  audits  open  to  assist  01  investigations,  the  most 
of  any  ORT  state. 

The  information  contained  in  the  audit  description  field  may  not  capture  all  instances  in  which  OAS  audits 
have  resulted  in  01  referrals,  or  all  audits  that  were  opened  at  the  request  of  OI.  It  does,  however,  provide 
quantitative  evidence  about  the  effects  of  the  greater  communication  and  collaboration  between  OAS  and 
OI  that  have  occurred  during  ORT. 


10    These  were  identified  based  on  the  pre.sence  of  -Assist  to  OI,"  ■  Ol/OGC,"  -Ol-Assist,"  -  Assist  OI."  -Ql  Assist, "  'OI  Support,"  "OI 
Asst."  "Asst  to  OI."  "Targei-OI,"  or  '  OI  Asist"  (sic)  in  the  audit  description  field 
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Exhibit  6-8:  ORT  Impact  Analyses 
Collaboration  Between  OAS  and  01 

Type  of  State 

Pre-ORT  period 

ORT  period 

ORT  state 

3 

15 

Non-ORT  state 

7 

7 

Source:  AIMS 

Collaboration  Between  OAS  and  HCFA 

In  15  audits,  OAS  joined  HCFA  and  state  certification  agencies  in  reviewing  Florida  skilled  nursing 
facilities  (Exhibit  6-9).  No  evidence  of  similar  collaboration  could  be  found  in  other  states  or  in  the  pre- 
ORT  period.  OAS  reports  that  questioned  costs  have  been  identified  as  a  result  of  these  audits,  the  results 
of  these  audits  were  not  recorded  in  AIMS  as  of  our  April  1997  data  cutoff  point. 


Exhibit  6-9:  ORT  Impact  Analyses 

Collaboration  Between  OAS  and  HCFA   

Type  of  State  Pre-ORT  period  ORT  period  

ORT  state  0  15 

Non-ORT  state  0  0 

Note:  Audits  that  involved  collaboration  between  OAS  and  HCFA  were  identified  by  the  presence  of  "part  of  HCFA  survey 
team"  in  the  audit  description  field. 

Source:  AllViS  '  

6.2.3     Impact  of  ORT  on  Questioned  Costs  Identified  by  OAS 

OAS  audits  can  result  in  two  basic  types  of  savings  for  the  Medicare  program: 

•  Direct  recoveries  from  providers  due  to  the  disallowance  and  recovery  of  a  questioned  cost 
uncovered  by  an  OAS  audit 

•  Cost  avoidance  due  to  identification  and  correction  of  program  vulnerabilities. 

AIMS  does  not  report  either  of  these  types  of  savings.  With  respect  to  the  first  type.  AIMS  does  report 
the  questioned  costs  that  are  disallowed  by  management,  but  does  not  track  these  disallowances  to  actual 
collection  from  providers,  as  collection  of  these  funds  is  not  the  responsibility  of  OAS.  As  a  result,  the 
questioned  costs  reported  in  AIMS  are  properly  viewed  as  "'receivables  established,"  rather  than  actual 
recoveries.  They  are  our  best  proxy  for  actual  collected  savings,  but  they  are  not  savings  per  se. 

With  respect  to  the  second  type,  savings  due  to  cost  avoidance  and  program  savings  are  not  reported  in 
AIMS.  Our  only  estimates  of  cost  avoidance  and  program  savings  attributable  to  ORT  are  reflected  in 
our  analysis  of  sentinel  effects  associated  with  ORT  (see  Appendix  10),  and  these  analyses  are  in  no  way 
tied  explicitly  to  the  efforts  of  OAS,  as  distinct  from  the  efforts  of  others  under  ORT. 
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There  are  an  important  set  of  complexities  in  estimating  OAS  effects  on  ORT  savings."  Many  audits  that 
have  identified  questioned  costs  have  resulted  in  referrals  to  01  or  other  law  enforcement  agencies. 
Because  these  audits  have  been  effectively  placed  on  hold  until  the  criminal  investigation  is  complete,  no 
Final  Report  has  been  issued,  and  the  disallowances  identified  by  these  audits  are  not  reported  in  AIMS. 
As  of  March  1997,  OAS  reported  that  these  audits  had  identified  S81  million  in  questioned  charges. 
Because  we  have  no  information  on  any  questioned  costs  identified  by  OAS  audits  in  either  the  pre-ORT 
f)eriod  or  from  comparison  states,  we  do  not  include  these  potential  savings  in  our  estimates  of  the  impact 
of  ORT  on  questioned  costs  identified  by  OAS  audits  of  ORT  focus-area  providers.  These  savings  are 
substantial.  To  that  extent,  our  savings  estimates  are  very  conservative. 

Evaluation  of  the  impact  of  ORT  on  potential  recoveries  from  providers  is  complicated  by  the  relatively 
small  number  of  audits  that  result  in  the  identification  of  questioned  costs.  Across  demonstration  and 
comparison  states  in  the  ORT  and  pre-ORT  periods,  only  33  audits  (of  ORT  focus-area  providers)  resulted 
in  the  identification  of  questioned  costs  that  are  reported  in  AIMS.  Two  of  these  audits,  which  together 
represent  nearly  50%  of  the  total  disallowances  identified  by  these  audits,  cannot  be  unambiguously 
assigned  to  the  ORT  or  pre-ORT  periods.  Our  definition  of  ORT  counts  these  audits  as  pre-ORT,  but, 
since  major  audit  activity  took  place  during  the  demonstration  period,  AIMS  designates  both  of  these 
audits  as  ORT-related. 

Eight  audits  that  we  defined  as  ORT  audits  resulted  in  the  identification  of  questioned  costs  that  were 
reported  in  AIMS.  These  eight  audits  resulted  in  disallowances  of  S21.6  million.  Seven  of  the  eight 
involved  Florida  providers,  including  five  home  health  agencies  (questioned  costs  of  $6.6  million),  one 
hospice  ($14.8  million),  and  one  skilled  nursing  facility  (less  than  $0.1  million).  The  final  audit  was  an 
audit  of  a  California  SNF,  which  resulted  in  a  disallowance  of  $0.2  million. 

By  contrast,  in  the  pre-ORT  period,  five  audits  resulted  in  the  identification  of  questioned  costs.  One  audit 
that  we  considered  to  be  from  the  pre-ORT  period  (since  it  opened  in  1994)  was  designated  as  ORT- 
related  in  AIMS  (since  some  major  audit  activities  occurred  after  ORT  started).  This  audit  (of  a  Florida 
home  health  agency),  found  that  $25.9  million  did  not  meet  reimbursement  guidelines  during  the  audit 
period.  Because  the  agency  filed  for  bankruptcy  in  August  1995,  however,  very  little  of  this  amount  is 
expected  ultimately  to  be  recovered  by  Medicare. 

More  than  $14  million  of  the  SI  6.3  million  total  for  comparison  states  resulted  from  an  audit  of  one  large 
home  health  agency  based  in  Georgia.  The  audit,  which  opened  in  1993,  was  not  completed  until  1996. 
It  found  unallowable  G&A  charges  of  $14  million  during  the  audit  period,  including  expenses  for 
computer  software,  salaries,  owner's  compensation,  marketing,  and  promotional  activities.  This  provider 
was  also  the  subject  of  a  criminal  investigation,  and  it  is  not  known  how  much  of  the  $14  million  in 
disallowances  identified  by  the  audit  were  ultimately  recovered  by  the  government. 

A  simple  "difference  in  differences"  model,  based  on  the  figures  in  Exhibit  6-10,  suggests  that  ORT  had 
a  large  impact  on  the  identification  of  questioned  costs.  This  impact  is,  however,  due  mainly  to  the  large 
decrease  observed  across  comparison  states  than  to  the  effect  observed  in  demonstration  states.  For 
example,  using  our  definition  of  ORT,  ORT  was  associated  with  a  $20  million  increase  in  costs  identified 


1  1     Problems  of  this  kind  are  not  as  senous  in  our  analysis  of  cxses  (Appendix  5 ).  since,  among  other  reasons,  the  number  of  cases  with 
receivables  identified  is  so  much  larger  than  single-case  complexities  do  not  have  so  much  leverage  on  our  estimates 
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by  OAS  audits.  Our  estimated  impact  of  ORT  would  have  been  higher  if  we  had  counted  either  of  the  two 
audits  discussed  above  that  were  considered  ORT-related  by  OAS  but  not  by  our  definition  of  ORT. 

It  is  important  to  emphasize  a  point  noted  earlier:  disallowances  should  increase  as  Final  Reports  are 
issued  for  the  audits  that  have  been  suspended  pending  the  outcome  of  criminal  investigations.  Due  to 
problems  with  incomplete  data,  and  because  the  estimated  effect  is  based  on  only  the  small  number  of 
audits  that  had  questioned  costs  already  reported  in  AIMS,  this  estimate  should  not  be  considered  a  robust 
or  reliable  estimate  of  the  impact  of  ORT  on  disallowances  identified  by  OAS.  Such  an  estimate  will  be 
possible  only  on  a  later  date. 


Exhibit  6-10:  ORT  Impact  Analyses 

Questioned  Costs  Identified  by  OAS  Audits  of  ORT  Focus-Area  Providers 

Type  of  State                               Pre-ORT  period 

ORT  period 

ORT  state  $42,451,352 

$21,609,573 

Non-ORT  state                                 $1 6,321 ,705 

$488,867 

Note:  Includes  questioned  costs  identified  in  AIMS  only.  Based  on  the  definition  of  ORT  described  in  Section  6.1 

Source;  AIMS 

6.3    Resources  Diverted  to  Pursue  ORT  Audits:  The  Opportunity 
Cost  of  ORT 


Although  ORT  travel  funds  proved  invaluable  in  allowing  OAS  to  pursue  the  type  of  provider-specific 
audits  that  it  emphasized  during  ORT,  the  agency  was  only  able  to  add  a  few  additional  staff  as  a  result 
of  ORT-  even  as  ORT  was  associated  with  a  very  large  increase  in  staff  resources  devoted  to  audits  of 
ORT  focus-area  providers.  Something  had  to  give.  Given  the  limited  additional  staff  made  possible  by 
ORT,  some  diversion  of  resources  from  other  types  of  audits  was  necessary  in  order  to  perform  these 
audits. 

In  this  section,  we  estimate  what  this  diversion  was.  It  can  be  viewed  as  a  partial  estimate  of  the 
"opportunity  cost"  of  ORT.  The  opportunity  cost  of  a  decision  arises  because,  given  limited  resources, 
choosing  to  engage  in  one  activity  means  giving  up  something  else.  The  opportunity  cost  of  OAS's 
emphasis  during  ORT  on  audits  of  DME  suppliers,  home  health  agencies,  nursing  homes,  and  hospices 
was  the  audits  that  were  not  performed  because  of  the  change  in  emphasis. 

To  say  that  this  opportunity  cost  existed  is  not  to  criticize  the  allocation  of  increased  efforts  to  ORT-type 
audits.  That  allocation  is  a  policy  decision  -  it  was  ORT's  authoritative  purpose  that  the  allocation  take 
place.  But,  if  we  do  not  perform  some  estimate  of  the  opportunity  cost  of  the  ORT  audits,  it  will  be  made 
to  appear  that  all  of  the  sizeable  increases  in  audit  volume  associated  with  ORT  were  obtained  for  nothing- 
as  if  auditors  were  dumb  before  ORT  started  and  suddenly  became  smart.  We  believe  that  ORT  is  a 
smarter  system,  as  discussed  elsewhere,  but  that  it  is  also,  importantly,  an  expression  of  the  effects  of 
added  resources.  Hence  the  need  to  estimate  opportunity  costs  the  best  we  can. 
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For  the  analyses  described  in  this  section,  we  included  all  audits  that  opened  after  1990.  except  for  those 
that  were  canceled  or  had  zero  cumulative  staff  days.  The  resulting  file  contains  information  on  5.988  audits. 

6.3.1     Opportunity  Cost  of  ORT:  Impact  of  ORT  on  Audit  Volume,  Subject  Types  and  Requesting 
Agencies 

Audit  Volume  for  all  Types  of  Audits 

ORT  was  associated  with  an  increase  in  audit  volume,  even  when  considering  all  types  of  audits  (Exhibit 
6-11).  During  ORT,  504  audits  opened  across  the  demonstration  states,  30%  more  than  would  have 
opened  if  ORT  states  had  the  same  pre/post  change  as  comparison  states  (where  volume  was  3,227  in  the 
pre-ORT  period  and  762  during  ORT).  Across  all  auditee  types,  ORT  was  associated  with  higher  volume 
for  all  five  demonstration  states  (Exhibit  6-1 2).  The  largest  increase  occurred  in  Florida,  where  volume 
was  nearly  100%  higher  than  it  would  have  been  in  the  absence  of  ORT.  Illinois  had  the  smallest  increase. 

The  30%  increase  in  overall  volume  was  less  than  the  impact  of  ORT  on  audits  of  ORT  focus-area 
providers,  which  increased  by  nearly  600%  during  ORT  (see  Exhibit  6-1 ).  Part  of  the  opportunity  cost 
of  ORT  appeared  to  be  a  reduction  in  the  number  of  audits  of  HHS  Operating  Divisions,  state  agencies, 
health  centers,  and  non-profit  agencies  (Exhibit  6-13).  This  reduction  appears  to  reflect  the  natural  effects 
of  what  ORT  sought-  an  increase  in  provider-specific  audits. 

In  ORT  states,  the  volume  of  audits  of  HHS  Operating  Divisions  decreased  from  1 83  to  22.  In  every  ORT 
state  except  Texas,  the  number  of  HHS  Operating  Division  audits  fell  by  at  least  90%.  Due  to  especially 
sharp  declines  in  Texas  (38  to  2),  California  (61  to  10),  and  New  York  (100  to  18),  audits  of  state  agencies 
fell  by  more  than  80%  in  ORT  states.  Only  three  audits  of  health  centers  opened  during  ORT,  compared 
to  24  in  the  pre-ORT  period.  Audits  of  non-profit  organizations  (other  than  academic  institutions) 
declined  from  163  to  30. 

Requesting  Agency  for  All  Types  of  Audits 

ORT  was  associated  with  a  relative  increase  in  the  number  of  audits  that  were  self-initiated  (i.e.,  audits 
for  which  AIMS  identifies  OAS  as  the  requesting  agency).  Specifically,  while  the  proportion  of  self- 
initiated  audits  fell  from  44%  to  39%  across  the  ORT  states,  the  decline  was  larger  (from  45%  to  27%) 
across  comparison  states  (Exhibit  6-14). 

Overall,  ORT  had  no  effect  on  the  total  number  of  audits  (of  all  types)  requested  by  01.  While  the 
proportion  of  audits  requested  by  OI  doubled  in  from  2%  to  4%  in  demonstration  states,  an  even  larger 
increase  was  observed  in  comparison  states.  This  result  did  not  appear  to  be  the  result  of  "leakage"  of 
ORT  effects  into  comparison  states,  as  the  same  basic  trend  was  observed  for  comparison  states  in  ORT 
regions  and  comparison  states  in  non-ORT  regions.  The  percentage  of  audits  requested  by  the  Inspector 
General  doubled  in  ORT  states  (as  against  basically  no  change  in  comparison  states).  ORT  was  associated 
with  a  small  increase  in  the  percentage  of  audits  requested  by  OP  DIV  (which  is  most  often  HCFA.  at 
least  for  health-care  related  audits),  which  went  from  15.6%  in  the  pre-ORT  period  to  17.9%  during  ORT. 
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Exhibit  6-1 1 :  ORT  Impact  Analyses 
Audit  Volume  for  Audits  of  All  Types 
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Source:  AIMS  (Data  through  April  1997) 
Notes: 

1  Includes  ORT  auditee  types  only  (DME  supplier,  home  health  agency,  nursing  home,  hospice). 

2  Non-ORT  states  include  all  non-ORT  states  with  an  Ol  suboffice. 

3  Pre-ORT  period  includes  audits  opened  1990-94;  ORT  period  includes  audits  opened  in  1995-97. 
4.  Excludes  cancelled  audits. 
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Exhibit  6-12:  ORT  Impact  Analyses 
Audit  Volume  by  State  for  Audits  of  All  Types 
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Source:  AIMS  (Data  through  April  1997) 
Notes: 

1 .  Includes  ORT  auditee  types  only  (DME  supplier,  home  health  agency,  nursing  home,  hospice). 

2.  Non-ORT  states  include  all  non-ORT  states  with  an  Ol  suboffice. 

3.  Pre  ORT  period  includes  audits  opened  1990-94;  ORT  period  includes  audits  opened  In  1995-97. 

4.  Excludes  cancelled  audits. 
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Exhibit  6-13:  ORT  Impact  Analyses 

Opportunity  cost  of  ORT:  Auditee  type  for  all  types  of  audits 


ORT  states  Comparison  state 


Auditee  Type 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

HHS  operating  division 

183 

22 

525 

73 

HHS  staff  division 

65 

23 

114 

15 

State/county  agency 

286 

57 

405 

67 

Other  govt. 

29 

14 

105 

7 

Hospital 

52 

18 

205 

17 

ORT  focus  areas 

17 

116 

30 

30 

PRO 

18 

9 

85 

9 

Health  center 

24 

4T 

•J 

o 

Medicare  contractor 

107 

20 

269 

48 

Other  medical 

19 

50 

86 

31 

College 

309 

113 

822 

288 

Other  non-profit 

163 

30 

274 

56 

For-profit  organization 

58 

10 

121 

23 

Totals 

1495 

504 

3227 

762 

Note:  Canceled  audits,  audits  with  zero  cumulative  staff  days,  and  audit  records  established  for  accounting  purposes  are 
excluded.  Includes  audits  of  ORT  focus-area  providers  only. 

Source:  AIMS 
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Exhibit  6-14:  ORT  Impact  Analyses 

Impact  of  ORT  on  Audit  Characteristics:  Requesting  Agency 


Requesting  Agency 

ORT  states 

Comparison  state 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

OAS 

659  (44%) 

1 96  (39%) 

1462  (45%) 

209  (27%) 

OCFAA 

0  (0%) 

1  (0.2%) 

0  (0%) 

2  (0.3%) 

IG 

56  (4%) 

38  (8%) 

134  (4%) 

28  (4%) 

01 

28  (2%) 

21  (4%) 

41  (1%) 

31  (4%) 

OEI 

1  (0.7%) 

1  (0.2) 

2  (0.1%,) 

1  (0.1%) 

RIGA 

129  (9%) 

39  (8%) 

76  (2%) 

7(1%) 

PRO 

6  (0.4%) 

4(1%) 

23  (0.4%) 

8  (1%) 

Hotline 

3  (0.2%) 

2  (0.4%) 

4  (0.1%) 

1  (0.1%) 

Non-Federal 

0  (0%) 

0  (0%) 

19  (1%) 

8  (1%) 

OIG:  Other 

19  (1%) 

0  (0%) 

35  (1%) 

24  (3%) 

HHS  Secretary 

24  (2%) 

7(1%) 

59  (2%) 

1 5  (2%) 

OP  DIV* 

234  (16%) 

90  (18%) 

576  (18%) 

128  (17%) 

HHS:  Other 

14  (1%) 

6(1%) 

32  (1%) 

14  (2%) 

Assist  USAO 

1  (0.7%) 

5(1%) 

0  (0%) 

15  (2%) 

Other  Federal  Dept 

187  (13%) 

59  (12%) 

378  (12%) 

153  (20%) 

Congress 

26  (2%) 

1  (0.2%.) 

86  (3%) 

12  (2%) 

Other  Federal 

98  (7%) 

28  (6%) 

299  (9%) 

105  (14%) 

Private  citizen 

3  (0.2%) 

1  (0.2%) 

2  (0.6%) 

0  (0%) 

Note:  Canceled  audits,  audits  with  zero  cumulative  staff  days,  and  audit  records  established  for  accounting  purposes  are 

excluded. 

A:  Includes  HCFA,  Public  Health  Services,  Administration  for  Children  and  Families,  Office  of  the  Secretary,  Administration  on 
Aging,  Agency  for  Toxic  Substances  and  Disease  Registry,  and  Agency  for  Health  Care  Policy  and  Research. 

Source;  AIMS 


Against  these  changes  in  referral  source,  there  was  a  decrease  in  audits  initiated  by  Congress,  other 
(unspecified)  branches  of  OIG,  and  other  (unspecified)  Federal  agencies  In  ORT  states,  the  number  of 
audits  initiated  at  the  request  of  Congress  declined  from  26  to  1 .  Audits  performed  at  the  request  of  other 
(unspecified,  but  not  OCFAA,  IG,  01,  OEI,  or  RIGA)  OIG  branches  fell  from  19  to  zero.  In  comparison 
states,  the  proportion  of  audits  with  this  requestor  code  nearly  tripled.  The  proportion  of  audits  requested 
by  other  (unspecified)  federal  agencies  increased  from  9%  to  14%  in  comparison  states,  but  declined 
across  the  ORT  states. 

ORT  thus  clearly  seems  to  have  changed  the  propensity  if  certain  sources  to  refer  audits  to  ORT  areas,  or 
of  OAS  in  demonstration  states  to  follow-up  on  them.  For  example,  referrals  that  in  the  past  might  have 
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been  performed  by  auditors  in  ORT  states  may  have  been  reassigned  to  non-ORT  states/regions  -  wiiich 
would  account  for  the  large  increases  in  these  referrals  in  for  these  non-ORT-type  audits. 

6.3.2    Opportunity  Cost  of  ORT:  Impact  of  ORT  on  Output  of  Audits 

ORT  was  associated  with  a  large  increase  in  the  proportion  of  audits  with  a  work  output  code  of  assist 
audit  work.  Such  audits  include  those  where  OAS  assists  01  with  a  criminal  investigation.  In  ORT  states, 
the  proportion  of  audits  involving  audit  assist  work  increased  from  10%  to  16%  (Exhibit  6-15).  Consistent 
with  the  emphasis  during  ORT  on  provider-specific  audits,  the  proportion  of  audits  that  resulted  in  an 
Audit  Report  on  Grantee/Contractor  Operations  increased  from  1 8%  to  29%  in  demonstration  states.  The 
percentage  of  audits  that  involved  staff  analysis  but  no  formal  report  also  increased  sharply,  from  1  %  to 
6%. 


Exhibit  6-15: 

Opportunity  Cost  of  ORT:  Impact  of  ORT  on  Audit  Characteristics:  Worl<  Output 


ORT  states 

Comparison  state 

Work  Output 

Pre-ORT  period 

ORT  period 

Pre-ORT  period 

ORT  period 

Assist  audit  work 

145  (10%) 

81  (16%) 

233  (7%) 

64  (8%) 

Audit  report  on  departmental 
operations 

104  (7%) 

3  (1%) 

276  (9%) 

29  (4%) 

Audit  report  on 

grantee/contractor  operations 

272  (18%) 

145  (29%) 

653  (20%) 

165  (22%) 

Survey  work* 

107  (7%) 

15  (3%) 

295  (9%) 

15  (3%) 

Nationwide  reporting 

15(1%) 

6  (1%) 

15  (0.4%) 

6  (1%) 

Follow-up  report 

42  (3%) 

2  (0%) 

99  (3%) 

2  (0.4%) 

Contract/grant  closing 
statement  audit  report 

140  (9%) 

35  (7%) 

396  (12%) 

111  (15%) 

Direct/indirect  cost  report 

57  (4%) 

10  (2%) 

87  (3%) 

11  (1%) 

Recipient  capability 

80  (5%) 

7(1%) 

125  (4%) 

7(1%) 

Pre-award/cost  proposal  audit 
report 

238  (16%) 

65  (13%) 

578  (18%) 

217  (28%) 

Prepared  by  CPA  under 
contract 

24  (2%) 

15  (3%) 

29  (1%) 

7(1%) 

Staff  analysis  :  No  report 

10(1%) 

29  (6%) 

77  (2%) 

22  (3%) 

A:  Survey  refers  to  phase  three  of  the  audit  process.  All  audits  should  be  initially  coded  with  a  work  output  code  indicating  that 
a  report  will  be  issued,  but  if  an  audit  is  ended  after  the  survey  phase,  the  work  output  code  should  be  changed  to  "Survey 
work." 

Source;  AIMS   
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Associated  with  these  increases  in  the  output  of  audits  was  a  decrease  in  audits  that  resulted  in  audit 
reports  on  departmental  operations,  nationwide  audit  reports,  and  pre-award/cost  proposal  audit  reports. 
These  changes  can  be  viewed  as  an  opportunity  cost  of  changing  the  kind  of  audits  performed  by  OAS. 
Across  ORT  states,  the  number  of  audit  reports  on  departmental  operations  declined  from  104  to  3.  a  much 
larger  decrease  than  occurred  in  comparison  states.  In  ORT  states,  the  proportion  of  audits  that  resulted 
in  a  nationwide  audit  report  decreased  from  9%  to  7%.  The  proportion  of  this  type  of  audits  increased 
from  1 2%  to  1 5%  in  comparison  states.  The  proportion  of  pre-award/cost  proposal  audits  increased  sharply 
in  comparison  states,  from  18%  to  28%  of  total  volume.  In  ORT  states,  however,  the  proportion  of  audits 
resulting  in  this  type  of  output  declined  from  16%  to  13%. 

6.3.3    Opportunity  Cost  of  ORT:  Diversion  of  OAS  Staff  Resources 

Although  OAS  was  able  to  add  only  a  few  additional  staff  as  a  result  of  ORT,  the  demonstration  was 
associated  with  a  large  increase  in  staff  days  devoted  to  audits  of  all  types  (Exhibit  6-16).  Across  the 
demonstration  states,  34,205  cumulative  staff  days  were  devoted  to  ORT-period  audits,  compared  to 
43,270  in  the  pre-ORT  period'-.  In  comparison  states,  only  46%  as  many  staff  days  had  been  devoted  to 
audits  of  all  types  during  the  ORT  period  than  in  the  pre-ORT  period.  Based  on  the  time  trend  observed 
in  comparison  states,  ORT  was  associated  with  an  increase  of  14,157  staff  days  (per  year)  devoted  to 
audits  of  all  types. 

The  additional  staff  hired  with  ORT  funds  can  account  for  only  a  small  proportion  of  this  increase,  and 
it  is  not  clear  why  ORT  had  such  a  large  impact  on  total  staff  days.  Given  that  audits  were  assigned  to 
states  based  on  the  location  of  the  auditee  type,  a  potential  explanation  is  that  auditors  from  non-ORT 
states  diverted  their  efforts  to  audits  within  demonstration  states  or  to  take  over  audits  that  auditors  from 
ORT  states  ordinarily  would  have  performed. 


Exhibit  6-16: 

Diversion  of  OAS  Resources:  Cumulative  Staff  Days 

Type  of  State 

Staff  days  per  year  for 
audits  opened  in  the  pre- 
ORT  period 

Staff  days  per  year 
for  audits  opened 
during  ORT 

Ratio  of  staff  days 
for  ORT-period  to 
pre-ORT  period 

ORT  states 

43,270 

34,105 

0.788 

Non-ORT  states 

83,562 

38,563 

0.461 

Total 

126,832 

72,668 

Note:  Includes  audits  of  all  types.  Pre-ORT  period  includes  audits  opened  between  1990  and  1994 
audits  opened  in  1995,  1996,  and  1997 

ORT  period  includes 

Source:  AIMS 

To  investigate  whether  such  diversion  occurred,  we  performed  two  additional  comparisons.  First,  we 
compared  effects  observed  in  ORT  states  to  other  (non-ORT)  states  within  ORT  regions.  Second,  we 
compared  effects  observed  for  comparison  states  in  ORT  regions  (i.e.,  regions  containing  an  ORT  state) 
to  comparison  states  in  non-ORT  regions. 


12     Given  that  some  audits  are  still  pending,  the  ratio  of  post-  to  pre-ORT  staff  days  will  increase  as  additional  staff  time  is  devoted  to 
pending  audits,  it  is  not  appropriate  to  say  that  staff  days  in  ORT  slates  actually  decreased  dunng  ORT. 
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Comparisons  of  Audit  Volume  Within  ORT  Regions  and  Between  ORT  and  non-ORT  Regions 


Comparing  only  states  within  ORT  regions,  ORT  was  associated  with  a  43%  increase  in  overall  audit 
volume  (Exhibit  6-17).  During  ORT,  504  audits  were  opened  in  ORT  states,  compared  to  331  for 
comparison  states  within  ORT  regions.  Volume  in  the  pre-ORT  period  was  comparable  for  ORT  and 
(1,495  audits)  and  comparison  (1,407)  states. 

The  impact  on  audit  volume  during  ORT  was  virtually  identical  for  comparison  states  in  ORT  and  non- 
ORT  regions  (Exhibit  6-17).  In  non-ORT  regions,  total  audit  volume  declined  from  1,820  during  ORT 
to  43 1  in  the  pre-ORT  period,  a  76%  decrease.  Comparison  states  in  ORT-regions  also  experienced  a  76% 
decline  in  overall  audit  volume  during  the  ORT  period. 


Exhibit  6-17:  ORT  Impact  Analyses 

Diversion  of  OAS  Resources:  Volume  of  Audits  Opened  in  Comparison  States  in  ORT  and 
Non-ORT  Regions 

Region 

Pre-ORT  period 

ORT  period 

ORT  region 

1,495 

504 

Non-ORT  region 

1,407 

331 

Note:  Includes  audits  of  all  types.  Pre-ORT  period  includes  audits  opened  between  1990  and  1994.  ORT  period  includes 
audits  opened  in  1995,  1996,  and  1997 

Source:  AIMS 

Comparisons  of  Staff  Days  Within  ORT  Regions  and  Between  ORT  and  non-ORT  Regions 

Focusing  only  on  states  within  ORT  regions,  the  demonstration  was  associated  with  a  large  increase  in 
staff  days  devoted  to  audits  of  entities  within  ORT  states.  Between  1990  and  1994,  nearly  707c  more  staff 
days  (per  year)  were  devoted  to  ORT-state  audits  than  for  audits  in  comparison  states  (Exhibit  6-18). 
During  ORT,  186%  more  staff  days  were  devoted  to  ORT-state  audits.  The  figures  in  Exhibit  6-18 
suggest  that  ORT  was  associated  with  an  additional  13,983  staff  days  devoted  to  audits  of  all  types  within 
ORT  states,"  suggesting  the  possibility  of  considerable  diversion  of  staff  resources  to  focus  on  ORT-state 
audits. 

Any  diversion  that  occurred  did  not  appear  to  come  disproportionately  from  ORT  regions.  The  ratio  of 
pre-ORT  to  ORT-period  staff  days  was  identical  for  comparison  states  in  ORT  and  non-ORT  regions 
(Exhibit  6-19). 


13     If  ORT  states  had  the  same  pre-posi  lime  trend  as  companson  states,  then  1 9.904  staff  days  would  have  been  devoted  to  audits  of  all 
types  dunng  CRT 
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Exhibit  6-18: 

Diversion  of  OAS  Resources:  Cumulative  Staff  Days  Within  ORT  Regions 


Staff  days  per  year  for     Staff  days  per  year     Ratio  of  staff  days 
audits  opened  in  the  pre-    for  audits  opened      for  ORT-period  to 
Type  of  State  ORT  period  during  ORT  pre-ORT  period 

ORT  states  43,270  34,104  0.79 

Non-ORT  states  25,657  11,939  0.46 

Ratio  of  ORT  state  to  1 .68  2.86 

non-0  RT  state  staff  days 

Note:  Includes  audits  of  all  types.  Non-ORT  states  include  comparison  states  within  ORT  regions.  Pre-ORT  period  includes 
audits  opened  between  1990  and  1994.  ORT  period  includes  audits  opened  in  1995,  1996,  and  1997 

Source:  AIMS 


Exhibit  6-19: 

Diversion  of  OAS  Resources:  Cumulative  Staff  Days  Between  Comparison  States  in  ORT  and 
non-ORT  Regions  

Staff  days  per  year  for    Staff  days  per  year     Ratio  of  staff  days 
audits  opened  in  the     for  audits  opened      for  ORT-period  to 
Comparison  state  in:  pre-ORT  period  during  ORT  pre-ORT  period 

ORT  region  25,657  11,939  0.46 

Non-ORT  region  57,905  26,624  0.46 

Ratio  of  ORT  region  to  0.44  0.44 

non-ORT  region  staff  days 

Note:  Includes  audits  of  all  types.  Pre-ORT  period  includes  audits  opened  between  1990  and  1994.  ORT  period  includes 
audits  opened  in  1 995.  1 996,  and  1 997 

Source:  AIMS 


6.4  Conclusions 

Obviously,  based  on  data  in  tfie  preceding  section,  there  are  signs  that  the  increases  in  audit  volume 
observed  during  ORT  came,  in  part  at  least,  from  the  diversion  of  other  efforts.  However,  the  gains 
achieved  during  the  demonstration  were  large.  ORT  resulted  in  a  large  increase  in  the  number  of  audits 
of  ORT  focus-area  providers.  The  increase  in  volume  was  spread  across  all  five  demonstration  states,  and 
across  all  ORT  focus-areas.  Reflecting  the  changes  in  priority  due  to  ORT.  OAS  auditors  devoted  1 8.920 
staff  days  to  audit  of  ORT  focus-area  providers  during  the  demonstration  period,  compared  to  only  2.766 
staff  days  on  such  audits  between  1990  and  1994. 

In  addition,  there  is  evidence  that  ORT  was  also  associated  with  increased  collaboration  between  OAS  and 
its  ORT  partners; 
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•  Fifteen  ORT  audits  were  opened  to  assist  01  with  a  criminal  investigation. 

•  There  was  a  sharp  increase  in  audits  that  involved  audit  assist  work,  which  were  primarily  audits 
where  OAS  assisted  either  01  with  a  criminal  investigation  or  HCFA  as  part  of  the  surveys  of 
SNFs. 

ORT  was  also  associated  with  changes  in  the  characteristics  of  OAS  audits: 

•  During  ORT,  there  was  a  large  increase  in  the  volume  of  audits  related  to  grantee  or  contractor 
operations,  many  of  which  were  provider-specific  audits  of  cost  reports  or  claims. 

•  The  number  of  audits  related  to  program  compliance,  services  provided,  and  alleged  criminal 
activities  increased  during  ORT. 

AIMS  reports  the  questioned  costs  identified  by  OAS  audits  with  which  management  concurs,  but  it  does 
not  track  these  disallowances  through  to  actual  collection  from  providers.  Meanwhile,  many  audits  that 
may  ultimately  result  in  the  identification  of  questioned  charges  are  still  pending  due  to  referral  to  01  for 
potential  criminal  prosecution.  No  Final  Report  has  been  issued  for  these  audits,  and  their  potential 
savings  are  not  yet  recorded  in  AIMS.  Similarly,  we  have  no  information  on  potential  savings  for 
comparison  state  audits  that  have  been  referred  to  law  enforcement.  Because  only  a  small  number  of  ORT 
focus-area  audits  have  data  on  questioned  costs  recorded  in  AIMS,  it  is  difficult  to  evaluate  the  savings 
attributable  to  OAS  audit  activities,  and  we  can  make  no  reliable  estimate  of  the  impact  of  ORT  on 
questioned  costs  identified  by  OAS.  The  data  that  are  recorded  in  AIMS  do  show,  however,  disallowances 
of  S21 .6  million,  an  amount  that  is  likely  to  grow  as  more  definitive  data  accrue  in  future  months. 
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Appendix  7 
Findings  from  the  Survey  of  Cases 


7.0  Introduction 

Abt  Associates'  evaluation  of  Operation  Restore  Trust  (ORT)  has  been  based  from  the  beginning  on  a 
simple  premise:  that  the  ORT  intervention  is  complex  and  elaborately  confounded  and  therefore  requires 
multiple  evaluation  approaches  to  establish  reliable  understandings  of  what  ORT  has  and  has  not 
accomplished.  The  ultimate  objective  of  these  various  approaches  is  to  determine  if  differing  data  bases, 
methods,  and  analytic  approaches  tell  something  like  a  consistent  story.  To  the  degree  results  are  relatively 
consistent  across  all  these  different  paths  of  evaluation,  we  can  assert  our  conclusions  with  greater 
confidence.  If  results  conflict,  our  findings  necessarily  become  more  contingent.  In  an  approximate  way, 
these  differing  approaches  are  designed  to  provide  some  rough  tests  or  checks  on  each  other. 

This  appendix  (and  Appendix  8  that  follows)  is  a  concise  reflection  of  this  overall  evaluation  strategy. 
Appendix  7  discusses  in  detail  the  results  of  a  survey  of  cases  recently  completed.  The  need  for  this 
survey  is  not  obvious.  Abt  staff  have  done  extensive  case  study  work  in  the  ORT  regions,  comparison 
regions,  and  central  offices  of  the  ORT  partners.  The  interviews  and  documents  provided  in  site  visits  and 
other  case  study  efforts  have  certainly  given  us  an  overall,  qualitative  appraisal  of  the  effects  of  ORT  on 
cases.  Meanwhile,  Abt  has  also  done  detailed  quantitative  analyses  of  ORT  effects  on  cases,  based  on  the 
Case  Information  Management  System  (CIMS)  of  the  Office  of  Inspector  General  (OIG).  Why  are  we 
doing  yet  another  data  collection  and  analysis  effort  to  understand  ORT's  effects  on  audits  and  cases? 

7.1  Motivation  for  the  Survey  of  Cases 

Cases  are  a  key  window  through  which  to  view  a  wide  and  diverse  spectrum  of  ORT  effects.  Certain 
major  effects  ORT  sought  to  achieve  -  e.g.,  to  enhance  collaboration  -  were  designed  to  register  on  the 
identification  and  development  of  cases.'  We  therefore  have  special  reason  to  focus  on  the  character  and 
frequency  of  ORT  effects  on  these  basic  building  blocks  of  enforcement.  We  need  to  understand  how 
ORT  affected  case  identification,  development,  and  follow-up.  ORT  presumes  that  there  will  be  certain 
changes  in  how  cases  originate,  how  they  are  developed,  and  what  outcomes  they  have.  Specifically,  ORT 
presumes  that: 

•     Case  identification  will  occur  in  response  to: 
more  collaboration  efforts 


Of  course,  cases  are  not  the  only  object  of  ORT  eflbrts  For  example.  ORT  was  also  designed  to  affect  the  identification  and 
development  of  audits  by  the  Office  of  Audit  Services  in  OIG.  as  well  as  to  affect  the  operations  of  the  Agency  on  Aging  and  of  HCFA 
and  Its  contractors  Only  some  of  the  effects  on  these  other  agencies  would  yield  a  formal  case  But  ORT  funds  were  devoted  in  large 
pan  10  case  land  audit)  development  by  the  OIG.  and  changes  in  case  processes  were  a  pnncipai  target  of  the  ORT  approach  Thus, 
while  they  are  not  the  only  target  of  ORT.  cases  were  a  panicularly  important  object  of  ORT. 
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-  more  sophisticated  targeting  activities,  notably  the  greater  use  of  databases,  computers, 
and  analytic  resources. 

•  Case  development  will  lead  to  more  expansive  investigations,  investigations  that  are: 

-  less  likely  to  stop  with  identified  or  isolated  instances  of  wrongdoing 

-  more  likely  to  seek  out  additional  examples  of  the  same  kind  of  wrongdoing  by  the  same 
parties  or  others. 

•  Case  results  will  be  somewhat  different,  in  that: 

-  instances  of  enforcement  will  more  frequently  lead  to  efforts  to  prevent  future 
occurrences. 

-  for  example,  where  a  case  would  have  ended  with  a  conviction  or  negotiated  settlement 
before,  it  will  now  lead  to  discussions  with  policy  makers,  contractors,  and  others  to 
establish  ways  to  prevent  or  limit  reoccurrences. 

The  hypotheses  outlined  above  can  be  tested  using  proxy  variables  in  the  automated  datasets  to  which  we 
have  access  -  notably,  the  CIMS  data.  But  there  is  a  risk  that  the  automated  datasets  and  qualitative 
interview  data  will  not  reveal  certain  characteristics  we  need  to  measure.  For  example,  CIMS  tracks  OI 
cases  from  the  date  a  complaint  is  first  received  until  the  date  the  case  closes.  The  system  details  case 
sources,  subject  types  and  location,  dates  of  major  case  development  milestones,  investigative  techniques 
used,  and  final  outcomes.  While  this  data  source  is  the  foundation  for  quantitative  analyses  Abt  Associates 
has  done,  it  does  not  provide  enough  qualitative  information  for  us  to  analyze  how  ORT  has  affected 
certain  key  characteristics  of  cases.  Specifically: 

•  CIMS  does  not  provide  direct  information  on  critical  changes  suggested  in  our  interviews, 
such  as  increases  in  promising  new  forms  of  collaboration,  the  establishment  of  new  methods 
of  targeting  investigations,  the  availability  of  key  resources  (e.g.,  travel)  to  permit  more  far- 
reaching  case  development,  and  other  possible  effects. 

Attributes  of  case  complexity  (e.g.,  notably  large  suspects  or  complicted  issues)  are  not 
always  obvious  from  CIMS  descriptors." 

Certain  characteristics  of  case  sources  are  not  revealed  (e.g.,  the  referral  source  is  identified, 
but  not  the  method  of  detection,  so  that  certain  issues  critical  to  understanding  ORT's  effects 

-  e.g.,  an  expanded  use  of  data  analysis  and  computer  resources  -  may  go  unrevealed). 

The  immediate  consequences  of  the  case  are  well  reported  (e.g.,  monetary  recoveries  and 
criminal  penalties),  but  possible  second-order  consequences  (e.g..  expanded  investigation  to 
other  suspects,  efforts  to  change  policy  and  procedures,  etc.)  are  not  identified. 

For  these  richer  descriptive  purposes,  we  needed  to  have  somewhat  richer  data  on  cases,  before  and  after 
ORT. 


2      CIMS  does  identify  cases  wiih  muUiple  suhjecis  However,  il  is  difficull  to  establish  changes  in  the  characteristics  of  subjects  beyond  a 
few  general  descnptors  as  to  type  of  subject  and  case. 


Abt  Associates  Inc. 


Appendix  7:  Findings  from  the  Survey  of  Cases 


7-2 


Certain  other  parts  of  our  evaluation  (e.g.,  site  visit  interviews)  provide  qualitative  data  and  anecdotes  on 
all  of  these  issues.  But  for  some  purposes,  that  information  is  weak  documentation.  Consider,  for 
example,  that  our  interviews  yielded  anecdotal  descriptions  of  new  collaboration  in  a  variety  of  different 
ways  under  ORT.  But  from  those  descriptions,  it  was  not  possible  to  answer  some  of  the  most  basic, 
important  questions  about  such  collaboration:  e.g.,  how  prevalent  was  it,  before  and  after  ORT?  All  we 
could  say  was  that  there  appeared  to  be  "more"  collaboration  after  ORT. 

We  therefore  needed  some  way  to  do  a  more  systematic  accounting  of  how  cases  had  changed  under  ORT. 
The  conventional  ways  to  acquire  such  information  would  be  through  surveys  or  records  abstraction.  We 
opted  for  a  survey,  for  practical  reasons.  We  could  not  review  case  records  (as  a  ground  rule  of  the 
evaluation,  Abt  staff  are  not  permitted  to  see  case  records).  In  any  event,  records  abstraction  would  be 
expensive  and  require  familiarity  with  the  records  involved,  which  we  do  not  have.  To  get  case-specific 
information,  we  therefore  needed  to  sun'ey  investigators,  using  survey  responses  as  proxies,  as  it  were, 
for  records  abstraction.  We  felt  that,  if  we  could  document  a  few  concrete  details  about  a  large  number 
of  cases,  we  would  be  in  a  position  to  answer  basic  questions  about  the  prevalence  and  magnitude  of  ORT 
effects  that  could  not  be  answered  any  other  way. 

We  therefore  have  collected  a  few  qualitative  details  about  a  large  sample  of  pre-ORT  and  post-ORT 
cases.  Aggregating  these  details  in  a  case-level  analysis  permits  us  to  quantify  the  prevalence  of  certain 
key  case  characteristics,  before  and  after  ORT.  The  difference  in  prevalence  before  and  after  ORT  will 
provide  us  with  first-order  estimates  of  changes  in  case  attributes  that  occurred  under  the  influence  of 
ORT.  These  estimates  are  subject  to  a  variety  of  qualifications,  as  we  will  discuss;  but  they  provide  a  far 
more  discriminating  picture  of  the  effects  of  ORT  than  we  could  practically  obtain  from  any  other  source. 

7.1.1     Responsibility  for  the  Surveys 

The  case  survey  was  designed,  implemented,  and  analysed  by  Abt  Associates,  as  described  in  the  report 
that  follows.  However, we  obviously  could  not  survey  investigators  in  the  Office  of  the  Inspector  General 
without  significant  assistance  from  the  Office  of  Investigations  (01)  in  OIG.  We  appreciate  the  assistance 
01  gave  us  in  this  effort;  we  especially  would  like  to  thank  the  investigators  who  took  the  time  to  complete 
the  surveys.  Of  course,  Abt  Associates  is  entirely  responsible  for  the  interpretation  of  results  that  follows. 

7.2    Overview  of  Main  Findings 

The  survey  results  that  are  described  in  the  following  report  confirm  our  expectation  that  case-level  data 
would  add  significantly  to  our  evaluation.  In  particular,  our  survey  of  cases  shows  non-exclusion  cases 
to  have  changed  under  ORT  in  the  following  ways: 

•  Referrals:  A  slight  increase  in  state  (MFCU)  referral  sources  and  collaboration  with  state 
agencies  (MFCUs  and  survey/certification  agencies)  in  identifying  targets  for  investigation. 
But  there  was  no  notable  increase  in  referrals  from  identified  federal  referral  sources. 

•  Case  complexity  (e.g..  number  and  location  of  targeted  providers):  Some  increase  in  case 
complexity  on  one-fifth  of  the  cases. 
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Other  unusual  attributes:  A  small  increase  in  the  proportion  of  unusual  cases. 


Collaboration  on  case  development:  There  was  a  substantial  increase  in  the  proportion  of 
cases  with  some  kind  of  collaboration  in  case  development.  As  with  the  case 
initiation/referrals,  this  increase  after  ORT  came  primarily  from  collaboration  with  state  rather 
than  federal  agencies. 

•  Resources:  Post-ORT,  there  was  a  puzzling  decline  in  the  proportion  of  cases  requiring 
unusual  staff  resources;  but  there  were  increases  in  the  proportion  of  cases  requiring  unusual 
amounts  of  travel,  computer  resources,  and  (especially)  medical/legal/other  resources.  There 
was  a  slight  decline  in  the  proportion  of  cases  for  which  development  was  limited  by  the 
shortage  of  some  resource. 

•  Investigative  techniques:  There  was  only  a  slight  increase  after  ORT  in  the  proportion  of  cases 
using  notably  sophisticated  investigative  techniques. 

•  Case  outcomes:  There  was  a  surprising  increase  after  ORT  in  the  proportion  of  cases  with 
"positive  outcomes"  such  as  a  criminal  referral.  The  increase  was  surprising  in  that  the  post- 
ORT  cases  have  not  had  as  much  time  as  the  pre-ORT  cases  to  run  their  course. 

•  Case  follow-up:  There  were  major  increases  after  ORT  in  case  follow-up,  in  such  terms  as 
the  frequency  of  looidng  for  other  instances  of  similar  misbehavior,  making  written  policy 
recommendations,  and  having  meetings  with  outside  agencies  and  Medicare  contractors. 
There  were  also  indications  that  this  follow-up  was  taking  place  somewhat  sooner  than  in  pre- 
ORT  cases. 


•  What  difference  does  ORT  make  in  cases?:  When  asked  how  pre-ORT  cases  would  have  been 
different  under  ORT,  and  how  post-ORT  cases  would  have  been  different  before  ORT,  most 
respondents  emphasized  matters  concerning  resources  and  collaboration. 

Meanwhile,  there  are  few  changes  in  exclusion  cases,  from  pre-ORT  to  post-ORT. 

These  results  suggest  that  the  frequency  of  certain  desired  case  characteristics  did  indeed  change  under 
ORT.  Perhaps  the  most  surprising  negative  result  we  observe  is  the  indication  that  federal-level 
collaboration  did  not  increase  in  case  initiation  activities  and  development.  But  in  other  areas  -  the  effect 
of  resources,  the  quality  of  cases,  the  prevalence  of  case  follow-up  activities  -  these  results  are  a 
reasonable  approximation  of  what  ORT  sought  to  accomplish.  And  the  results  in  one  area  (case  outcomes) 
are  surprising,  and  in  a  second  area  (case  follow-up)  are  especially  large  and  impressive. 


Our  discussion  of  the  survey  of  cases  is  organized  as  follows: 


7.3  Design  and  methodology,  including  a  description  of  the  pre/post  design  we  have  used  in  our 
survey,  the  biases  in  that  design,  how  we  have  assigned  cases  to  pre-  and  post-ORT  periods, 
and  other  issues. 
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7.4  Data  collection,  including  descriptions  of  our  sample  design,  our  survey  instrument,  and  our 
procedures  for  conducting  the  survey  (particularly  procedures  concerning  confidentiality). 

7.5  Presentation  of  case-specific  results:  non-exclusion  cases. 

7.6  Presentation  of  case-specific  results:  exclusion  cases. 

7.7  Presentation  of  responses  to  our  investigator-specific  questions. 

7.3  Methodology 

Our  basic  method  for  analyzing  the  survey  data  on  cases  is  by  using  a  single-difference,  pre-post  model. 
In  this  section,  we  explain  that  choice  and  certain  decisions  that  influenced  how  it  should  be  applied. 

7.3.1     Use  of  a  Pre/Post  Model 

As  we  have  argued  elsewhere  (Abt  Associates,  1996a),  the  ideal  model  for  estimating  demonstration 
effects  is  a  so-called  "double-difference"  model.  The  simple  logic  of  that  model  is  to  estimate 
demonstration  effects  by  comparing  demonstration  results  to  pre-demonstration  results  in  demonstration 
areas,  as  adjusted  for  secular  trends  estimated  by  looking  at  the  pre/post  changes  in  selected  comparison 
areas.  In  this  fashion,  the  model  provides  some  control  for  secular  trends  coincident  with  the 
demonstration  that  (in  a  pure  pre/post  design)  might  improperly  be  attributed  to  the  demonstration. 

Notwithstanding  the  theoretical  appeal  of  the  double-difference  model,  however,  we  did  not  feel  we  could 
apply  such  a  model  in  this  case.  The  key  reason:  it  would  greatly  increase  the  burden  of  the  survey, 
particularly  in  terms  of  the  number  of  investigators  and  separate  field  offices  that  would  have  to  be 
involved  in  the  survey.  This  increase  in  burden  would  be  because: 

•     The  sample  sizes  would  have  to  be  large  enough  to  permit  multivariate  estimation  of  the 
model's  parameters. 

The  inclusion  of  non-ORT  regions  would  bring  a  large  group  of  non-ORT  agents  into  the 
survey.' 

We  did  not  feel  we  could  expand  the  survey  in  either  respect,  much  less  both.  We  should  note  that  this 
decision  was  ours  (OI  did  not  insist  on  it).  It  was  our  judgment  that  the  simpler  survey  ultimately  adopted 
imposed  as  much  burden,  and  required  as  much  cooperation  and  support,  as  we  could  practically  ask  01 
to  consider.  As  a  result,  we  opted  to  limit  the  sample  size  (see  Section  7,4  below)  and  not  to  use 
comparison  states  in  our  design.  Instead,  we  chose  to  use  a  simple  pre/post  model  in  the  survey  of  cases. 


There  was  one  way  possibly  to  reduce  this  second  form  of  burden.  Specifically,  if  the  comparison  states  were  all  selected  from  within 
the  ORT  regions,  the  survey  could  to  some  degree  be  restncted  to  ORT  participants.  But  to  the  degree  this  restriction  was  effective, 
estimates  of  ORT  effects  would  likely  be  biased  by:  a)  likely  ■  leakage"  of  positive  ORT  effects  to  non-ORT  cases,  since  the  companson 
cases  would  in  this  event  be  cases  done  by  the  same  agents  who  handled  the  demonstration  cases:  and/or  b)  constraints  on  non-ORT 
cases  caused  by  possible  within-region  diversions  of  effon  and  resources  to  ORT  cases.  In  our  judgment,  these  biases  were  too  serious 
to  warrant  designing  the  survey  using  comparison  states  from  the  ORT  regions. 


Abt  Associates  Inc. 


Appendix  7:  Findings  from  the  Survey  of  Cases 


7-5 


Even  though  we  did  not  use  a  double-difference  model  in  our  design  for  the  survey  of  cases,  it  is  important 
to  keep  the  logic  of  double  differences  in  mind,  as  the  cost  of  not  using  comparison  states  in  our  design 
(the  loss  of  control  for  secular  trends)  is  one  of  the  most  important  qualifications  on  our  analysis  of  results. 
We  at  least  need  to  be  aware  that  the  pre/post  results  we  observe  in  the  ORT  states  could  be  the  result  of 
many  other  factors  common  to  otherwise  similar.  non-ORT  states. 

7.3.2    Determining  What  Is  a  "Pre"  Case  and  What  Is  a  "Post"  Case 

To  apply  a  pre/post  model,  we  obviously  need  to  understand  how  cases  are  assigned  to  the  "pre"  and 
"post"  categories.  The  problem  is  conceptually  a  bit  messy,  since  cases  typically  have  such  long  lives: 
according  to  our  analysis  of  CIMS,  the  median  case  lasted  3.5  years  just  before  ORT  began,  with  the 
longest  case  lasting  almost  6.0  years.^  Given  the  length  of  cases,  some  pre-ORT  cases  must  have 
continued  long  after  ORT  began.  The  question  then  becomes:  are  such  cases  pre-ORT  or  post-ORT  cases? 

To  answer  this  question,  we  define  three  types  of  cases,  based  on  the  date  the  case  opened  and  the  date 
of  major  case  development  milestones: 

•  "Pure  pre-ORT  case"  -  A  pure  pre-ORT  case  is  one  that  opened  before  ORT  and  either 
closed  or  was  referred  for  prosecution  prior  to  the  start  of  ORT  (i.e.,  before  April  1 .  1995). 
These  are  cases  that  were  initiated  and  more  or  less  fully  developed  before  ORT  started.^ 

•  "Bridge  case"  -  A  bridge  case  is  one  that  opened  before  ORT  started,  but  was  either  still 
pending,  or  was  referred  for  prosecution  or  closed  during  the  ORT  period.  These  cases  were 
initiated  before  ORT,  but  had  varying  amounts  of  development  after  ORT  began. 

•  "Pure  ORT  case"  -  A  pure  ORT  case  is  a  one  that  opened  on  or  after  April  1,  1995  (i.e., 
during  the  demonstration  period).  For  these  cases,  both  initiation  and  development  occurred 
after  ORT  began. 

It  is  obvious  how  pure  pre-ORT  and  pure  post-ORT  cases  should  be  assigned.  Bridge  cases  present  the 
major  difficulty.  The  key  issue  is  how  we  should  assign  these  cases,  based  on  the  information  available 
to  us.  Even  if  ORT  had  no  practical  effects  on  such  cases,  there  are  endless  possible  different  ways  ORT 
could  have  affected  these  cases,  even  if  it  failed  to  do  so.  If,  for  example,  the  discernible  effects  of  ORT 
on  bridge  cases  are  zero  (to  take  an  extreme  possibility),  that  result  may  simply  describe  a  limit  in  ORT's 


This  calculation  is  based  on  all  ORT  subject-type,  non-exclusion  cases  nationwide,  in  the  penod  1991  -  1994  Note  that  the  distiibution 
is  skewed:  these  cases  last  an  average  of  just  over  20  months  The  average  is  lower  because  a  relativeK  large  number  of  cases  have  very 
short  lives. 

Please  note  that  this  definition  of  a  pre-ORT  case  differs  from  the  definition  used  in  our  recent  Interim  Repon.  In  the  Interim  Repon.  a 
case  had  to  open  and  clone  before  ORT  began,  in  order  for  the  case  to  be  counted  as  a  pure  pre-ORT  case.  For  this  survey,  by  contrast, 
we  are  relaxing  that  definition  somewhat.  In  this  Final  Report,  the  case  does  not  have  to  close;  it  need  only  have  reached  the  stage  of 
referral  to  the  prosecutor.  This  new  definition  assumes  that  most  Ol  work  on  a  case  will  generally  have  been  completed  by  the  time  of 
such  referral.  While  in  many  cases  that  is  not  true,  it  is  a  more  realistic  definition  of  a  cutoff  point  for  ORT  effects  than  would  be  a  later 
cutoff  (e  g.,  the  point  at  which  the  case  is  closed)  In  effect,  we  assume  that,  after  referral  to  a  prosecutor,  ORT-specific  effects  on  the 
case  are  much  less  likely. 
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reach,  rather  than  something  intrinsic  in  the  cases  themselves.^  As  a  result,  we  need  to  segregate  these 
cases  from  pure  pre-ORT  cases  and  pure  post-ORT  cases  for  analytical  purposes,  where  ORT  might  have 
had  an  effect. 

We  have  adopted  a  practical  approach  that  takes  account  of  the  limited  data  available  to  us:  we  consider 
bridge  cases  as  "pre-ORT"  cases  for  survey  questions  dealing  with  case  initiation  and  identification  onh: 
For  all  other  questions,  we  report  results  for  bridge  cases  separately  from  pure  pre-ORT  and  pure  post- 
ORT  cases. ^ 

7.3.3    Analyzing  Exclusion  Cases  Separately 

One  further  consideration  in  our  design  was  whether  or  not  to  treat  all  CEMS-identified  cases  (i.e.,  all 
matters  with  a  separate  case  number)  as  homogeneous,  or  to  consider  analyzing  some  kinds  of  cases 
separately  from  others.  In  particular,  "exclusion"  cases  presented  the  most  important  issues.  The  reason 
is  that  these  cases  usually  represented  administrative  follow-up  work  to  enforcement  actions  by  others 
outside  DHHS/  OIG  -  e.g.,  follow-up  work  to  exclude  providers  from  Medicare,  after  state  cases  had 
assessed  Medicaid  liability  against  providers.  Such  follow-up  work  was  thought  to  be  different  in  kind 
from  a  case  that  was  developed  by  01  from  the  start. 

As  a  result,  while  our  sample  sizes  are  not  so  large  that  we  can  elaborately  stratify  responses,  this 
rudimentary  division  did  seem  helpful,  for  two  principal  reasons: 

1 .  The  number  of  exclusion  cases  increased  substantially  under  ORT  (see  our  analysis  of  CIMS 
data  in  Appendix  5  of  this  Final  Report).  As  a  result,  the  post-ORT  cohort  of  "cases" 
included  a  disproportionate  number  of  exclusion  cases.  Analyzing  the  exclusion  cases 
separately  would  prevent  this  change  in  the  mix  of  case  types  from  pre-determining  the 
analysis  of  changes  in  case  characteristics. 

2.  For  many  of  these  exclusion  cases,  agents  suggested  that  the  assignment  of  case  numbers  was 
an  administrative  convenience  for  minor  amounts  of  follow-up  work.  These  modest  activities 
were  depicted  as  being  different  in  kind  from  a  case  that  was  initiated  and  fully  developed  in- 
house.^ 

Accordingly,  we  separately  report  results  for  exclusion  cases  in  the  analyses  that  follow. 


6  For  an  anal  ogous  reason,  we  cannot  (without  more  information!  take  an  agent's  claim  that  a  particular  bridge  ca.se  'was  a  pre-ORT  case" 
at  face  value,  since  thai  could  simply  be  a  way  of  saying  that  everything  important  m  the  case's  life  occurred  before  ORT  -  which  could 
be  a  measure  of  the  liinit  of  CRT's  effects,  rather  than  something  intnnsic  to  the  case  itself. 

7  We  did  consider  other  rules,  such  as  requinng  a  bndge  case  to  be  open  a  panicular  number  of  ORT  months  (or  a  particular  percentage 
of  Its  total  life  dunng  ORT).  The  problem  wiih  a  rule  of  this  sort  is  thai  any  given  choice  would  involve  debatable  assumptions  about 
the  limits  of  ORT's  effects.  So  we  would  still  need  to  do  some  kmd  of  sensitivity  analysis  to  make  sure  that  the  choice  did  not  pre- 
determine our  results.  In  other  words,  we  would  need  to  isolate  the  bridge  cases  for  separate  analysis.  We  thus  decided  to  keep  those 
cases  separate  for  most  pans  of  the  analysis,  in  order  lo  simplify  our  analysis. 

8  Our  survey  confirms  this  point.  See  section  7.6  of  the  text 
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7.3.4    Principal  Limitations  of  the  Analysis 


Our  survey  of  cases  provides  far  more  discriminating  information  than  is  available  from  any  other  source. 
It  is  worth  noting  the  limitations  of  this  data,  however: 

•  Post-ORT  cases  are  "censored"  due  to  the  cutoff  point  of  the  data  -  As  discussed  in  detail  in 
our  analysis  of  CIMS  (Appendix  5).  ORT-period  data  are  censored,  since  -  by  comparison 
to  pre-ORT  and  bridge  cases  -  these  ORT  cases  have  not  existed  long  enough  for  all  of  their 
results  to  accrue.  That  means  that,  for  survey  questions  concerning  development  and 
outcomes,  the  ORT  cases  are  an  unavoidably  biased  sample  -  the  true  ultimate  outcomes  are 
likely  to  be  understated  in  the  snapshot  taken  by  our  survey.  Accordingly,  in  our 
interpretation  of  questions  about  case  development  and  outcomes,  we  need  to  be  mindful  that 
ORT  cases  are  incomplete  in  a  temporal  sense.' 

•  Some  of  the  most  fruitful  cases  -  those  that  have  gone  to  grand  juries  or  are  at  other 
particularly  sensitive  stages  -  have  been  disproportionately  deleted  from  the  post-ORT  sample 
-  We  had  to  delete  a  number  of  post-ORT  cases  from  our  sample  due  to  various  kinds  of 
confidentiality  requirements,  such  as  grand  jury  restrictions  on  discussing  the  case  with  any 
outside  party  and  other  similar  problems.  No  pre-ORT  cases  were  deleted  for  this  reason. 
It  seems  reasonable  to  assume  that  cases  in  these  sensitive  stages  are  among  the  more  fruitful 
or  productive  cases.  Disproportionately  excluding  them  from  the  post-ORT  sample  will  tend 
to  reduce  our  estimates  of  ORT  effects. 

•  The  survey  measures  frequency,  not  intensity  -  We  know  from  interviews  that  OI  field  staff 
consider  ORT  an  important  change  in  how  they  do  business.  The  question  is,  what  forms  of 
ORT  effects  are  consistent  with  this  sense  that  ORT  is  an  important  change?  We  can  imagine 
at  least  two  ways  that  ORT  might  affect  cases.  Both  ways  are:  1)  consistent  with  our 
interview  data,  but  2)  not  captured  equally  well  by  our  survey.  First,  if  ORT  had  effects  on 
a  lot  of  cases,  our  survey  would  likely  do  well  at  capturing  those  effects,  since  in  that  event 
measures  of  prevalence  (e.g.,  percent  of  cases  using  computer-driven  identification  of 
subjects)  will  show  a  noticeable  change  under  ORT.  However,  if  ORT  has  effects  (small  or 
large)  on  only  a  few  cases,  it  will  leave  many  or  even  most  other  cases  untouched  in  terms  of 
major  characteristics  being  surveyed.  Meanwhile,  in  this  latter  case,  the  effects  on  a  few  cases 
could  be  substantial  or  defining. '°  Notwithstanding  that  these  intense  effects  involve  only  a 
relatively  small  proportion  of  cases,  such  effects  would  represent  an  important  change  in  how 
01  does  business.  But  our  counts  of  survey  responses  would  show  little  change,  since  our 
survey  does  not  provide  much  indication  of  the  intensity  of  effects. 


9  ll  IS  accordingly  remarkable  lhat  our  survey  nonetheless  shows  major  mcreases  in  positive  case  outcomes  and  follow-up  for  post-ORT 
cases.  Indeed,  these  results  are  the  most  important  results  of  the  sur\ey. 

10  For  example,  under  ORT  cenain  kinds  of  cases  that  were  rare  or  non-existent  before  ORT  may  be  brought:  e  g.,  self-referred  cases  that 
were  identified  by  computer  analysis;  or  cases  at  a  long  distance  thai  are  possible  to  investigate  using  ORT  travel  funds. 
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In  other  words,  our  survey  is  better  at  capturing  some  effects  on  a  lot  of  cases  than  large 
effects  on  a  few  cases  -  or,  more  generally,  at  capturing  notable  changes  in  frequency  than 
notable  changes  in  intensity. 

In  view  of  these  limitations,  we  need  to  be  careful  in  how  we  interpret  the  survey  data.  The  important 
point  is  that  -  even  in  view  of  these  intrinsic  limitations  -  the  survey  data  are  far  more  discriminating  than 
all  other  available  data  and  show  major  ORT  effects.  As  our  analysis  below  will  show,  these  data  permit 
us  to  understand  cases  in  a  much  richer  way  than  would  otherwise  be  possible. 

7.4    Data  Collection 

7.4.1     Sample  Design 

To  minimize  problems  associated  with  recall  error  and  turnover  among  01  agents,  we  restricted  the  survey 
sample  to  cases  that  opened  after  January  1,  1993.  To  ensure  that  cases  included  in  the  sample  would 
have  at  least  some  time  for  development  (eight  months),  we  only  included  cases  that  had  opened  by 
August  31.  1996. 

Our  original  sample  design  assumed  a  total  sample  of  180  cases.  These  were  generally  to  be  distributed 
as  follows: 

•  By  period  -  This  allocation  was  a  subjective  judgment,  the  purpose  of  which  was  to  ensure 
adequate  numbers  of  pre-ORT,  bridge,  and  post-ORT  cases.  We  decided  on  a  split  of  one- 
half  post-ORT  cases  and  one-half  pre-ORT/bridge  cases.  The  latter  category  was  further 
divided:  two-thirds  pre-ORT,  one-third  bridge  cases. 

•  By  state:  -  This  allocation  was  determined  by  the  relative  case  volumes  for  each  state  over 
the  1993  -  1996  period,  as  identified  in  CIMS. 

As  this  design  was  implemented,  there  were  a  number  of  different  changes  caused  by  practical  problems: 

•  Three  cases  were  deleted  (e.g.,  because  we  could  not  connect  the  case  to  a  field  office),  and 
177  case  surveys  were  sent  out.  Ordinarily,  we  would  have  replenished  these  three  cases;  but 
coincidentally,  we  received  complaints  of  burden  at  this  time.  Since  the  result  of  the  ensuing 
discussions  was  to  cut  the  sample  size,  we  did  not  replenish  these  three  cases. 

•  Two  regions  with  large  samples  -  New  York  and  Texas  -  indicated  that  they  felt 
overburdened  by  the  number  of  questionnaires  they  had  received.  In  response,  we  agreed  to 
reduce  the  sample  sizes  for  these  two  states  to  32  cases  each  (including  pilot  surveys,  which 
had  already  been  completed  and  returned).  These  deletions  reduced  the  sample  to  142. 

•  Not  all  of  the  142  surveys  for  cases  remaining  in  the  sample  could  be  completed  -  e.g., 
because  of  grand  jury  restrictions  or  the  absence  of  the  lead  investigator  (staff  turnover).  This 
led  to  sample  attrition.  We  replenished  the  sample  with  randomly  selected  substitute  cases. 
However,  because  of  a  shortage  of  pre-ORT  cases  in  some  areas  (notably  California,  where 
an  investigator  had  retired),  the  proportion  of  bridge  cases  increased. 
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•  Replenishment  was  a  special  concern  for  pre-ORT  cases,  as  problems  of  attrition  affected  pre- 
ORT  cases  disproportionately.  As  the  survey  deadline  neared.  we  added  1 1  pre-ORT  cases 
from  our  random  list.  The  cases  were  selected  in  order  from  our  random  list;  however,  given 
the  tight  timing,  we  contacted  RIGs  directly  and  discussed  which  cases  (going  down  the  list) 
could  be  completed  by  investigators,  as  we  did  not  have  time  to  send  substitute  surveys  and 
wait  to  find  out  if  the  survey  could  be  completed  (e.g.,  to  find  out  if  the  investigator 
responsible  for  the  case  was  available  to  complete  a  survey  about  the  case). 

In  the  end,  having  designed  a  sample  of  180,  we  received  128  usable  surveys.  The  basic  differences 
between  our  original  design  and  our  final  usable  count  were  as  shown  in  Exhibits  7- la  and  7- lb.  The  non- 
exclusion  cases  are  our  greatest  concern.  As  is  apparent  from  Exhibit  7- la,  the  proportions  of  usable 
surveys  vary  by  period,  so  that  our  sample  of  usable  surveys  has  more  post-ORT  and  (especially)  bridge 
cases  than  in  the  original  design.  Exhibit  7-2  shows  the  final  sample  proportions,  by  state,  period,  and 
subject  type. 


Exhibit  7-1  a 

Survey  of  Cases:  Total  Non-Exclusion  Cases  by  Period  and  by  Disposition 


Pre-ORT 

Bridge 

Post-ORT 

Total 

Total  Non-Exclusion  Cases  in  Original 
Sample  ^ 

31 

29 

48 

108 

Net  Change  after  Attrition  and 
Replenishment 

(1) 

3 

(8) 

6 

Completed  Surveys  Received 

30 

32 

40 

102 

Not  Usable' 

11 

1 

3 

15 

Usable  (N) 

19 

31 

37 

87 

Usable  Surveys  as  %  of 
Original  Sample 

61% 

1 07% 

77% 

81% 

'  Reasons  cases  were  not  usable  include:  case  pending  with  a  grand  jury,  investigator  unavailable,  case  file  cannot  be 

located,  CIN  is  an  administrative  account,  and  limited  Ol  involvement. 

Source:  Abt  Associates,  1997 
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Exhibit  7-1  b 


Survey  of  Cases:  Total  Exclusion  Cases  by  Period  and  by  Disposition 


Pre-ORT 

Bridge 

ORT 

Total 

Total  Exclusion  Cases  in  Original  Sample 

29 

0 

43 

72 

Net  Change  after  Attrition  and 
Replenishment 

(9) 

1 

(16) 

(25) 

Received 

20 

1 

27 

48 

Not  Usable' 

7 

0 

0 

7 

Usable  (N) 

13 

1 

27 

47 

Usable  Surveys  as  %  of 
Original  Design 

45% 

n/a 

63% 

57% 

Reasons  include:  case  pending  with  a  grand  jury,  investigator  unavailable,  case  file  cannot  be  located,  CIN  is  an 
administrative  account,  and  limited  Ol  involvement. 


Source:  Abt  Associates,  1997 


Exhibit  7-2a 

Survey  of  Cases:  Usuable  Non-Exclusion  Surveys  by  State,  Subject  Type  and  Period 

Period 


State 

Subject  Type 

Pre-ORT 

Bridge 

ORT 

Total 

California 

DME 

2 

0 

4 

6 

HHA 

1 

2 

6 

9 

SNF 

0 

4 

0 

4 

Florida 

DME 

5 

6 

13 

24 

HHA 

0 

0 

0 

0 

SNF 

0 

0 

1 

1 

Illinois 

DME 

1 

1 

2 

4 

HHA 

2 

1 

0 

3 

SNF 

0 

0 

0 

0 

New  Yorl< 

DME 

8 

7 

4 

19 

HHA 

0 

0 

2 

2 

SNF 

0 

1 

0 

1 

Texas 

DME 

0 

2 

2 

4 

HHA 

0 

7 

2 

9 

SNF 

0 

0 

1 

1 

Total  DME 

16 

16 

25 

57 

Total  HHA 

3 

10 

10 

23 

Total  SNF 

0 

5 

2 

7 

Total 

19 

31 

37 

87 

Source:  Abt  Associates,  1997 
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Exhibit  7-2b 

Survey  of  Cases:  Usuable  Exclusion  Surveys  by  State,  Subject  Type  and  Period 

Period 


Olal6 

ouDjecx  1  ype 

rre-Uri  1 

Bridge 

ORT 

Total 

lif  nrnifl 

DMF 

IVI  ^ 

1 

1 

n 

R 

f 

HHA 

n 

1 

i 
1 

olNr 

n 

u 

U 

•4 

1 

1 

FInriHa 

DMF 

n 

w 

n 

n 
U 

HHA 

0 

w 

n 

n 

u 

u 

(J 

rv 
U 

0 

lllinniQ 

Hill  l(.^lO 

DMF 

1 

1 

o 
o 

HHA 

u 

ft 

CMC 

U 

u 

U 

0 

Mpva;  Ynrk 

1  >IC  VV    I  liJI  (\ 

DMF 

O 

1 

O 

c. 

c 

3 

HHA 

1 

o 

bNr 

2 

0 

4 

6 

Texas 

DME 

0 

0 

0 

0 

HHA 

1 

0 

3 

4 

SNF 

5 

0 

7 

12 

Total  DME 

4 

1 

10 

15 

Total  HHA 

2 

0 

5 

7 

Total  SNF 

7 

0 

12 

19 

Total 

13 

1 

27 

41 

Source:  Abt  Associates,  1997 


We  thus  were  forced  to  adjust  our  stratified  randomized  sample  to  accommodate  a  series  of  understandable 
practical  difficulties.  We  attempted  to  preserve  the  random  character  of  the  cases  sampled.  But  given 
the  relatively  small  N  in  relation  to  the  attrition,  it  is  necessary  to  be  cautious  in  thinking  about  the 
systematic  biases  that  may  have  been  introduced.  Our  analyses  of  attrition  (data  not  shown)  fail  to  reveal 
any  statistically  significant  biases  in  the  attrition,  except  with  respect  to  period  (pre-ORT  cases  were  lost 
disproportionately  from  the  sample)  -  that  is,  we  do  not  find  important  within-period  biases.  The  biases 
we  note  across  periods  is  readily  dealt  with,  however,  by  analyzing  data  separately  by  periods,  which  we 
do  for  all  purposes  except  case  initiation. 

7.4.2    Survey  Instrument 

A  copy  of  the  survey  instrument  and  cover  letter  are  included  in  Attachment  1  to  this  appendix.  The 
questionnaire  was  revised  on  several  occasions-  notably,  after  feedback  from  01  in  January  1997  and  after 
the  pilot  test  in  April  1997.  The  survey  form  in  Attachment  1  is  the  version  that  was  distributed  to 
investigators  when  we  fielded  the  full  survey. 
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The  survey  asks  for  a  series  of  qualitative  answers  on  certain  key  areas  of  interest  in  light  of  the  ORT 
intervention.  The  survey  questions  in  paraphrased  form  are  as  shown  on  Exhibit  7-3.  As  the  exhibit 
suggests,  we  used  the  opportunity  of  this  case-specific  survey  to  ask  some  agent-specific  questions. 


Exhibit  7-3 

Survey  of  Cases: 

Summary  of  Survey  Questions 

Question  Type 

and  Number 

Question 

CASE-SPECIFIC  QUESTIONS 

1 

What  prompted  the  investigation? 

2 

Was  the  initiation  of  this  case  unusual  or  atypical  (e.g.,  in  terms  of  the  number  of  providers 

targeted,  their  location,  the  complexity  of  the  scheme,  or  the  complexity  of  the  investigation 

itseii) 

3 

Was  there  anything  else  unusual  about  this  case? 

4 

With  reSDect  to  identification  of  thp  ca^ip  tarnpt  Hiri  vnii  h;5\/p  anu  nntahip  mllahoratinn  with 

federal,  state,  and  other  agencies? 

5 

With  respect  to  development  of  the  case,  did  you  have  any  notable  collaboration  with 

federal,  state,  and  other  agencies? 

6 

Did  case  development  require  any  unusual  amount  of  resources  (e.g.,  staff,  travel,  etc.)? 

7 

Was  case  development  limited  by  the  shortage  of  any  particular  type  of  resource? 

8 

Approximately  how  much  overnight  travel  did  this  case  require? 

9 

Were  any  unusual  or  notably  sophisticated  investigative  techniques  used? 

10 

Have  certain  important  outcomes  occurred  (e.g.,  referral  to  a  prosecutor,  recoupment. 

formal  charges,  etc.)? 

11 

Were  any  follow-up  actions  taken  as  a  result  of  the  case  (e.g.,  written  policy 

recommendations,  exploration  to  find  others  exhibiting  a  similar  pattern  of  misbehavior, 

etc.)? 

12 

Did  these  follow-up  actions  occur  the  case  was  closed  or  referred  to  another  agency? 

13 

How  would  this  case  have  been  different  under  ORT?  or 

How  would  this  case  have  been  different  before  ORT?" 

INVESTIGATOR-SPECIFIC  QUESTIONS 

14 

What  in  ORT  has  worked  well? 

15 

What  in  ORT  has  not  worked  well? 

16a. 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt? 

16b 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopt? 

Source:  Abt  Associates  Survey  of  Cases,  1997. 


1 1      Two  versions  of  the  questionnaire,  pre  and  post,  were  created  to  facilitate  asking  two  different  versions  of  this  hypothetical  question 


Abt  Associates  Inc. 


Appendix  7:  Findings  from  the  Survey  of  Cases 


7-13 


To  minimize  burden,  we  limited  the  complete  survey  to  5  pages.  Most  questions  were  close-ended  (i.e., 
the  respondent  was  required  only  to  check  a  box,  rather  than  to  write  out  a  detailed  answer).  For  a  few 
questions,  we  provided  space  for  very  brief,  open-ended  responses. 

7.4.3  Pilot  Test 

After  receiving  feedback  from  01  on  our  initial  proposal  for  the  survey  of  cases,  we  incorporated 
comments  into  the  pilot  version  of  the  instrument  and  accompanying  letters.  In  April  1997,  ten  pilot  test 
survey  packets  were  mailed  to  Regional  Inspector  Generals  (RIGs),  who  distributed  the  packets  to 
investigators.  Based  on  the  responses  we  received,  we  revised  the  survey  slightly.  The  main  change  we 
made  was  to  develop  check-box  responses  for  several  questions,  to  simplify  completion  of  the  survey. 

7.4.4  Conduct  of  the  Survey 

We  fielded  the  survey  in  early  May  1997.  At  that  time,  bundles  of  surveys  were  sent  to  the  appropriate 
RIGs.  In  turn,  the  RIGs  distributed  each  individual  survey  package  to  the  lead  investigator  responsible 
for  the  associated  case.  Each  survey  package  contained  the  following: 

•  a  letter  from  Abt  Associates, 

•  a  questionnaire  with  pre-printed  case  information  (CIN#,  state/region,  subject  type,  date 
open,  date  close,  and  period),  and 

•  a  pre-addressed,  postage-paid  return  envelope. 

We  then  tracked  the  survey  responses  over  the  weeks  that  followed.  Weekly  tracking  reports  were  faxed 
to  01,  which  followed  up  on  surveys  that  had  not  yet  been  returned.  Surveys  were  received  up  to  June  30, 
1997. 

If  a  survey  could  not  be  completed,  RIGs  provided  investigators  with  a  substitute  survey  from  a  random 
list,  following  procedures  specified  by  Abt.  These  substitute  surveys  had  the  same  period  and  state  as  the 
original  surveys;  however,  due  to  sample  and  other  limitations,  subject  types  could  vary  from  our  original 
design. 

A  Microsoft  Access  data  entry  system  was  developed  for  the  purposes  of  entering  survey  data.  For  open- 
ended  questions,  a  coding  map  was  developed.  Responses  were  coded  accordingly,  and  then  entered  into 
the  database. 

7.4.5  Confidentiality 

The  confidentiality  of  the  survey  of  cases  was  a  major  concern  to  01  and  to  individual  investigators. 
Particularly  on  open  cases,  it  was  obviously  important  to  ensure  that  our  data  collection  and  report  did  not 
create  any  problems  for  OI's  investigative  work.  To  that  end,  our  case  survey  report: 

•  Does  not  include  multiple  fields  of  any  case-specific  survey  responses,  as  might  allow  a 
reader  to  stitch  together  a  series  of  features  to  infer  the  identity  of  a  case  or  group  of 
cases. 
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•  Uses  aggregates  of  cases  for  its  analyses,  rather  than  identifying  any  specific  aspect  of 
any  specific  case. 

•  Does  not  quote  any  case-specific  open-ended  answers. 

•  Was  reviewed  by  01  Central  Office  before  being  circulated  to  others,  to  catch  any 
possible,  inadvertent  disclosures. [OI  review  pending] 

These  procedures  were  designed  to  ensure  that  no  case  identities  were  revealed  in  our  report.  Even  with 
these  assurances,  several  surveys  were  returned  incomplete,  with  notes  attached  indicating  that  the  cases 
were  still  pending  before  a  grand  jury  and  therefore  subject  to  strict  confidentiality  requirements.  These 
cases  were  eliminated  from  our  sample. 

7.5    Case-specific  Survey  Results:  Non-exclusion  Cases 

The  following  section  presents  our  case-specific  survey  results  for  the  non-exclusion  cases.  For 
convenience,  we  proceed  more  or  less  consecutively  through  the  survey  questions,  but  have  grouped 
clusters  of  questions  together  when  they  address  the  same  or  similar  issues. 

1.       Changes  in  the  initiation  of  cases  (Questions  1  and  4) 

Exhibit  7-4  shows  results  of  a  query  about  how  the  case  was  initiated:  specifically,  what  the  referral 
sources  were.  As  can  be  seen  from  the  exhibit,  the  identification  of  subjects  of  cases  did  not  change  much 
from  pre-ORT  to  post-ORT  cases.'"  Percentages  are  little  changed,  except  perhaps  in  two  respects:  a 
somewhat  higher  proportion  of  MFCU-referred  cases  after  ORT,  and  a  somewhat  lower  proportion  of  01 
referrals  from  outside  the  field  office. 


Exhibit  7-4 

Survey  of  Cases:  Referral  Sources 

Non-Exclusion  Exclusion 


Referred  from 

Pre 

Post 

Pre 

Post 

OAS 

0% 

3% 

0% 

0% 

0!  in  another  region 

6% 

0% 

0% 

4% 

HCFA 

6% 

3% 

0% 

0% 

HCFA  contractor 

42% 

41% 

0% 

0% 

MFCU  or  other  state  agency 

0% 

11% 

93% 

93% 

Other 

44% 

41% 

0% 

0% 

Don't  Know 

2% 

3% 

7% 

4% 

Total 

100% 
N  =  50 

100% 
N  =  37 

100% 
N  =  14 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding. 


Source:  Abt  Associates.  1997 


12     li  IS  worth  noiing  again  that,  for  the  identification  phase  of  cases,  bndge  cases  are  considered  pre-ORT  ca.ses 
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Exhibit  7-5a  shows  a  somewhat  different  dimension  of  case  initiation:  responses  to  a  question  about 
whether  or  not  any  notable  collaboration  occurred  in  the  identification  of  the  subject(s)  for  investigation. 
Collaboration  is  up  under  ORT:  the  cases  with  collaboration  with  at  least  one  non-OI  agency  increased 
from  70%  to  over  80%.  Exhibit  7-5b  suggests  that  most  of  this  increase  in  collaboration  was  generated 
with  «o/i-0/?7  agencies:  specifically,  with  state  agencies  of  some  kind.  Most  of  these  are  MFCUs  and 
state  survey/certification  agencies  (data  not  shown).  That  result  is  consistent  with  our  site  visits,  with 
Exhibit  7-4  above,  and  with  what  we  found  in  our  analysis  of  CIMS  (see  Appendix  5).  However,  we  do 
not  see  any  increase  in  collaboration  in  case  identification  from  ORT  partners  and  related  entities  - 
notably,  from  HCFA  or  its  contractors.  Some  of  the  HCFA  influence  may  be  picked  up  in  the  increase 
in  collaboration  from  state  survey/certification  agencies. 


Exhibit  7-5a 

Survey  of  Cases:  Notable  Collaboration  in  the  Identification  of  Case 
Subjects 

Non-Exclusion  Exclusion 


Response 

Pre 

Post 

Pre 

Post 

No  Notable  Collaboration 

30% 

19% 

36% 

22% 

At  least  some  Collaboration 

70% 

81% 

64% 

78% 

Total 

100% 
N  =50 

100% 
N  =37 

100% 
N  =14 

100% 
N  =27 

Note:  Percentages  may  not  sum  to  100%  due  to  rounding. 
Source:  Abt  Associates.  1997  


Exhibit  7-5b 

Survey  of  Cases:  Collaboration  in  Identification  of  those  Cases  with  Notable 
Collaboration 

Non-Exclusion  Exclusion 


Collaboration  With 

Pre 

Post 

Pre 

Post 

State  Only 

3% 

23% 

89% 

95% 

Federal  Only 

69% 

53% 

11% 

0% 

Federal  and  State 

17% 

13% 

0% 

5% 

Other  Only 

9% 

7% 

0% 

0% 

Federal  and  Other 

0% 

3% 

0% 

0% 

Federal,  State  and  Other 

3% 

0% 

0% 

0% 

Total 

100% 
N  =  35 

100% 
N  =  30 

100% 
N  =  9 

100% 
N  =  21 

Note;  Percentages  may  not  sum  to  100%  due  to  rounding. 
Source:  Abt  Associates.  1997 


While  these  differences  are  generally  in  the  expected  direction  for  ORT  effects,  they  are  small.  We  could 
as  easily  focus  on  what  has  not  changed,  because  it  appears  that  -  in  the  crude  terms  of  our  questions  - 
most  cases  have  changed  little  in  character. 
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2. 


Changes  in  the  complexity  of  cases  (Question  2) 


In  general,  when  characterizing  the  complexity  of  post-ORT  cases,  investigators  did  not  see  a  major 
change  from  the  typical  pre-ORT  case  in  almost  80<7c  of  the  cases  (there  were  changes  in  complexity 
indicated  for  only  22%  of  the  cases)  (Exhibit  7-6a).  But  the  balance  of  cases  were  viewed  as  atypical  from 
the  pre-ORT  norm.  Post-ORT  cases  were  slightly  different  in  the  number  of  targeted  providers,  the 
location  of  the  targeted  providers,  the  complexity  of  the  scheme  being  investigated,  and  the  complexity 
of  the  investigation  (Exhibit  7-6b).  This  change  is  consistent  with  our  interview  data  -  e.g.,  pointing  to 
the  influence  of  added  ORT  resources  on  the  reach  of  investigations  (see  discussion  of  answers  to  our 
resources  questions  below). 


Exhibit  7-6a 

Survey  of  Cases:  Were  there  Changes  in  Case  Complexity  Compared 

to  Typical  Pre-ORT  Cases? 

Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No 

21% 

52% 

54% 

15% 

100% 

93% 

Yes 

11% 

23% 

22% 

0% 

0% 

0% 

Don't  Know 

0% 

0% 

8% 

0% 

0% 

0% 

Not  Applicable 

69% 

26% 

8% 

77% 

0% 

4% 

No  Answer 

0% 

0% 

8% 

8% 

0% 

4% 

Total 

100% 

100% 

100% 

100% 

100% 

100% 

N  =  19 

N  =  31 

N  =  37 

N  =  13 

N  =  1 

N  =  27 

Note:       Percentages  may  not  sum  to  1 00%  due  to  rounding. 

Source:  Abt  Associates,  1997 

Exhibit  7-6b 

Survey  of  Cases:  Unusual  Initiation  Compared  to  Pre-ORT  Cases? 

Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No 

21% 

52% 

54% 

15% 

100% 

93% 

Yes-  Number  of  providers 

5% 

16% 

11% 

0% 

0% 

0% 

Yes-  Location  of  providers 

5% 

7% 

11% 

0% 

0% 

0% 

Yes-  Alleged  scheme  of  infraction 

5% 

7% 

14% 

0% 

0% 

0% 

Yes-  Complexity  of  investigation 

0% 

3% 

8% 

0% 

0% 

0% 

Yes-  Other 

5% 

3% 

5% 

0% 

0% 

0% 

Don't  Know 

0% 

0% 

8% 

0% 

0% 

0% 

Not  Applicable 

69% 

26% 

8% 

77% 

0% 

4% 

No  Answer 

0% 

0% 

8% 

8% 

0% 

4% 

Total 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

N  =  19 

N  =  31 

N  =  37 

N  =  13 

N  =  1 

N  =  27 

Note:     Respondents  were  encouraged  to  cite  all  the  ways  in  which  the  case  was  unusual, 
so  rows  are  not  mutually  exclusive  and  percentages  sum  to  more  than  100%. 

Source:  Abt  Associates,  1997 
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3. 


Changes  in  other  unusual  case  features  (Question  3) 


Exhibit  7-7  shows  investigators'  overall  estimation  if  there  were  anything  else  unusual  about  the  cases 
surveyed,  beyond  the  complexity  of  the  cases  and  how  they  were  initiated.  The  exhibit  shows  an  increase 
(from  16%  to  247c)  in  the  proportion  of  cases  viewed  as  unusual,  from  pre-  to  post-ORT. 


Exhibit  7-7 

Survey  of  Cases:  Changes  in  Other  Unusual  Case  Features 


Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No 

79% 

84% 

73% 

100% 

100% 

93% 

Yes 

16% 

16% 

24% 

0% 

0% 

7% 

No  Answer 

5% 

0% 

3% 

0% 

0% 

0% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:      Respondents  were  encouraged  to  cite  all  the  ways  in  which  the  case  was  unusual,  so  rows 
are  not  mutually  exclusive  and  percentages  sum  to  more  than  100%. 


Source:  Abt  Associates,  1997 

4.       Case  development  1:  Collaboration  (Question  5) 

The  first  issue  in  case  development  was  to  review  whether  or  not  collaboration  had  increased  in  any 
important  way.  Exhibit  7-8  suggests  that  collaboration  did  indeed  increase  substantially:  there  was  some 
notable  collaboration  in  almost  79%  of  the  pre-ORT  cases,  but  over  89%  in  the  post-ORT  cases.  The 
increased  collaboration  in  case  development  was  largely  with  state  agencies  -  largely,  MFCUs  and 
survey/certification  agencies  (data  not  shown). 


Exhibit  7-8 


Survey  of  Cases:  Notable  Collaboration  in  Case  Development 


Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No  notable  collaboration 

21% 

39% 

11% 

69% 

100% 

70% 

At  least  some  collaboration 

79% 

61% 

89% 

31% 

0% 

30% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding. 


Source:  Abt  Associates.  1997 


5.       Case  development  2:  Resources  (Questions  6,  7,  and  8) 

Exhibit  7-9  suggests  that  post-ORT  cases  are  indeed  different  in  terms  of  the  resources  they  use,  but  these 
differences  are  only  partially  what  we  would  predict.  The  proportion  of  cases  with  no  unusual  resource 
requirements  in  fact  is  little  changed  under  ORT  (roughly  three-quarters  of  all  cases,  as  is  true  for  pre-ORT 
and  bridge  cases).  But  there  are  notable  changes  in  other  areas: 
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Exhibit  7-9 

Survey  of  Cases:  Unusual  Resources  Required  for  Case  Development 


Non-Exclusion 

Exclusion 

Resource  Type 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

Staff  Time 

26% 

23% 

14% 

0% 

0% 

22% 

Travel  Funds 

5% 

7% 

8% 

0% 

0% 

0% 

Computer  Resources 

5% 

16% 

14% 

0% 

0% 

15% 

Medical  Review 

5% 

0% 

8% 

0% 

0% 

0% 

Communication  Costs 

0% 

0% 

0% 

0% 

0% 

4% 

Legal  Resources 

0% 

0% 

5% 

0% 

0% 

0% 

Other 

0% 

3% 

3% 

0% 

0% 

0% 

None  of  the  Above 

74% 

71% 

76% 

77% 

100% 

74% 

No  Answer 

0% 

3% 

3% 

23% 

0% 

4% 

Total 

n/a 
N  =  19 

n/a 
N  =  31 

n/a 
N  =  37 

n/a 
N  =  13 

n/a 
N  =  1 

n/a 
N  =  27 

Note:       Respondents  were  encouraged  to  cite  all  resources  required,  so  rows  are  not  mutually  exclusive  and  percentages  sum 
to  more  than  1 00%. 


Source:  Abt  Associates,  1997 

•  Staff  time  -  There  is  a  very  large  drop  in  the  proportion  of  cases  that  required  unusual 
staff  resources:  from  23%  -26%  for  pre-ORT  and  bridge  cases  to  14%  for  post-ORT 
cases.  It  is  not  clear  why  this  would  happen,  given  the  staff  increases  that  accompanied 
ORT  and  the  (admittedly  slight)  increases  in  case  complexity  after  ORT  (see  discussion 
of  Question  2  above).  One  possible  explanation  is  that,  on  average.  post-ORT  cases  came 
to  investigators  in  better-developed  form,  thus  reducing  the  unusual  resource  needs  after 
the  cases  were  opened.  Another  possible  explanation  is  that  unusual  amounts  of  time 
from  investigators  were  not  required  as  often  when: 

-  Constraints  in  certain  non-staff  resources  were  eased  in  such  key  areas  as  travel, 
computers,  and  medical  review  (see  discussion  of  medical  review  resources 
below);'-'  and 

-  Case  development  processes  were  somewhat  more  efficient,  through  greater 
collaboration. 

But  these  are  all  conjectures,  and  we  are  unable  to  say  with  confidence  why  individual 
cases  required  unusual  staff  resources  in  the  pre-ORT  and  bridge  periods  to  a  much 
greater  extent  than  the  post-ORT  period. 

•  Travel  -  A  slightly  higher  proportion  of  post-ORT  cases  (8%)  used  unusual  amounts  of 
travel  compared  to  pre-ORT  cases  (5%).  Bridge  cases  were  in  between  these  two  limits. 


1.^     Note  lhai  the  use  ot  medical  review  resources  would  likely  show  up  in  our  survey  as  "medical  review"  resources,  rather  than  "staff 
resources. 
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possibly  reflecting  a  slight  influence  of  ORT  on  the  bridge  cases.  However,  in  terms  of 
the  amount  of  overnight  travel  (Exhibit  7-10),  the  post-ORT  cases  (14%)  show  only  a 
slight  increase  from  pre-ORT  cases  (1 19c)  in  the  proportion  with  "some"  travel  (<  20 
days  of  overnight  travel)  or  a  "lot"  of  overnight  travel  (20  days  or  more).  The  puzzling 
result  is  bridge  cases,  for  which  a  much  higher  proportion  (29%)  had  some  or  a  lot  of 
travel.  If  ORT  had  this  effect  on  the  bridge  cases,  we  would  expect  the  ORT  effect  on 
post-ORT  cases  to  be  much  larger. 

Computer  resources  -  14%  of  post-ORT  cases  used  unusual  amounts  of  computer 
resources  in  case  development,  compared  to  only  5%  in  pre-ORT  cases.  For  reasons  that 
are  unclear,  1 6%  of  the  bridge  cases  used  unusual  amounts  of  computer  resources,  again 
possibly  reflecting  a  slight  influence  of  ORT  on  the  bridge  cases,  although  we  cannot  be 
sure. 

Medical  review,  legal,  and  other  resources  -  Almost  16%  of  the  post-ORT  cases  made 
special  use  of  medical  review,  legal,  and  other  resources,  compared  to  almost  none  of  the 
pre-ORT  and  bridge  cases.  In  other  words,  these  post-ORT  cases  were  categorically 
different  than  the  pre-ORT  cases  -  e.g.,  in  using  Peer  Review  Organizations  (PROs)  to 
assist  in  cases  by  providing  medical  review  resources,  a  practice  essentially  without 
precedent  before  ORT.  (Note  that  our  site  interviews  confirm  this  difference;  see 
Appendix  1.) 


Exhibit  7-10 

Survey  of  Cases:  Amount  of  Overnight  Travel  Required  by  the  Case 


hJon-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

None 

90% 

68% 

84% 

100% 

100% 

96% 

Some  (e.g.,  less  than  20  days) 

11% 

26% 

14% 

0% 

0% 

0% 

A  lot  (e.g.,  20  days  or  more) 

0% 

3% 

0% 

0% 

0% 

0% 

No  Answer 

0% 

3% 

3% 

0% 

0% 

4% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding. 


Source:  Abt  Associates.  1997 

Meanwhile,  the  proportion  of  cases  for  which  case  development  was  limited  by  the  shortage  of  some 
resource  declined  somewhat:  32%  of  pre-ORT  and  bridge  cases  were  said  to  be  limited  by  a  resource 
shortage,  compared  to  27%  of  post-ORT  cases  (Exhibit  7-11). 

Overall,  we  thus  see  evidence  that  post-ORT  cases  made  somewhat  greater  use  of  computer  resources  and 
travel  than  their  pre-ORT  counterparts,  made  lesser  staff  demands  (a  bit  of  a  puzzle),  and  made  unique 
uses  of  medical  review/legal/other  resources.  In  pattern,  if  not  extent  (almost  all  of  the  differences  are 
small),  these  results  comport  well  with  simple  predictions  based  on  what  ORT  was  trying  to  accomplish. 
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Exhibit  7-11 


Survey  of  Cases:  Development  Limited  by  Shortage  of  Some  Kind  of  Resource 


Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No 

68% 

65% 

68% 

92% 

100% 

85% 

Yes 

32% 

32% 

27% 

0% 

0% 

11% 

No  Answer 

0% 

3% 

5% 

8% 

0% 

4% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding. 


Source:  Abt  Associates,  1997 


The  principal  exceptions  to  this  summary  appraisal  are  the  absence  of  indications  of  more  intensive 
staffing  in  the  post-ORT  cases  and  indications  of  more  extensive  travel  in  the  bridge  cases  compared  to 
the  post-ORT  cases. 

6.       Case  development  3:  Unusual  or  notably  sophisticated  investigative  techniques, 
compared  to  the  typical  pre-ORT  case  (Question  9) 

This  question  was  designed  to  elicit  information  about  more  frequent  use  of  sophisticated  investigative 
techniques  under  ORT,  compared  to  the  typical  pre-ORT  case.  The  meaning  of  "unusual"  or  "notably 
sophisticated"  was  not  defined  (short  definitions  we  could  construct  did  not  seem  to  improve  on  the 
common  understandings  of  the  original  terms).  In  any  event,  the  results  suggest  a  slight  increase  in  cases 
using  unusual  investigative  techniques,  from  5%  pre-ORT  to  1 1  %  post-ORT  (Exhibit  7-12). 


Exhibit  7-12 

Survey  of  Cases:  Unusual  or  Notably  Sophisticated  Investigative  Techniques,  Compared  to  Pre- 
ORT  Cases 


Non-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

No  Unusual  Techniques 

37% 

77% 

73% 

46% 

100% 

96% 

Some  Unusual  Techniques 

5% 

7% 

11% 

0% 

0% 

0% 

Don't  Know 

0% 

0% 

3% 

0% 

0% 

0% 

Not  Applicable 

58% 

13% 

11% 

54% 

0% 

4% 

No  Answer 

0% 

3% 

3% 

0% 

0% 

0% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding. 


Source:  Abt  Associates,  1997 

7.       Case  outcomes  (Question  10) 
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One  aspiration  of  ORT  was  to  enhance  the  quality  of  cases;  higher  quahty  cases  would  mean  more 
frequent  positive  outcomes.  This  area  was  problematic  to  survey,  since  (compared  to  pre-ORT  cases)  post- 
ORT  cases  have  not  had  a  comparable  time  to  yield  positive  outcomes  and  have  a  disproportionate  share 
of  cases  excluded  due  to  grand  jury  secrecy  and  other  concerns.  What  is  notable  in  our  results  is  that, 
notwithstanding  these  problems.  post-ORT  cases  do  show  a  somewhat  higher  rate  of  positive  outcome. 
We  can  define  "positive  outcome"  to  include  any  case  that  results  in: 

•  A  referral  to  the  U.S.  Attorney,  the  FBI,  or  other  law  enforcement  agency, 

•  Filing  of  formal  civil  or  criminal  charges, 

•  Exclusion, 

•  Recoupment,  or 

•  Other  (e.g.,  provider  bankruptcy). 

Exhibit  7-13  suggests  that  there  has  been  a  noticeable  increase  in  cases  with  a  positive  outcome:  from 
53%  (pre-ORT)  to  73%  (post-ORT).  The  bridge  cases  have  a  somewhat  higher  figure  for  positive 
outcomes  (81%),  which  could  reflect  their  longer  gestation  period  as  of  the  time  of  the  survey  or  the 
influences  of  ORT,  or  both. 


Exhibit  7-13 

Survey  of  Cases:  Significant  Case  Outcomes 


Mon-Exclusion 

Exclusion 

Response 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

Total  with  a  positive  outcome ' 

53% 

81%> 

73% 

100% 

100% 

96% 

Total  without  a  positive  outcome 

47% 

19% 

27% 

0% 

0% 

4% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  13 

100% 
N  =  1 

100% 
N  =  27 

Note:       1 )  Positive  outcomes  include:  referral  of  some  kind,  formal  criminal  or  civil  charges,  exclusion  and  recoupment. 
2)  Reasons  for  cases  without  a  positive  outcome  include:  case  closed,  no  outcome,  no  answer,  don't  know. 


Percentages  may  not  sum  100%  due  to  rounding. 
Source:  Abt  Associates,  1997 

Compared  to  the  pre-ORT  cases,  the  post-ORT  increase  in  proportion  of  cases  with  positive  outcomes  is 
one  of  the  most  remarkable  results  of  the  entire  survey  (the  other  remarkable  result  is  in  case  follow-up, 
as  discussed  immediately  below).  This  result  is  the  closest  thing  we  have  in  the  survey  to  a  summary 
measure  of  whether  or  not  ORT  effects  -  whatever  they  are  -  made  an  ultimate  difference  in  case 
outcomes.  Already,  it  appears,  the  ORT  cases  show  signs  of  higher  quality  than  their  pre-ORT 
predecessors  -  notwithstanding  that  we  would  expect  the  post-ORT  figure  to  be  biased  downward, 
compared  to  figures  for  pre-ORT  and  bridge  cases. 

8.       Case  follow-up  (Questions  1 1  and  12) 
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Another  area  where  ORT  sought  to  improve  on  traditional  practices  was  in  what  might  be  termed  "case 
follow-up."  In  the  past,  it  was  felt,  enforcement  process  failed  to  take  a  given  case  and  follow-up  on  its 
implications  in  a  proactive  way.  Thus,  discovery'  of  one  provider  engaging  in  a  particular  pattern  of  fraud 
did  not  lead  often  enough  to  1)  more  thorough  exploration  (e.g.,  searching  for  other  providers  doing  the 
same  thing),  or  2)  preventive  efforts  (e.g.,  contacting  HCFA  to  discuss  the  need  for  policy  changes). 

The  results  on  this  score  are  striking.  As  shown  on  Exhibit  7-14,  there  were  large  increases  in  the 
proportion  of  cases  with  the  following  kinds  of  follow-up:'^ 


Exhibit  7-14 

Survey  of  Cases:  Follow-up  Actions 

Non-Exclusion 

Exclusion 

roiiow-up  ACiion 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

Commenced  investigations  of 

^imilpr  nrnviHprQ 

OIIIIMCll  IJrWVlUdO 

11% 

10% 

38% 

8% 

0% 

4% 

Submitted  written 
recommendations  for  policy 
changes 

5% 

0% 

11% 

0% 

0% 

4% 

Met  with  agency  outside  OIG 

11% 

16% 

27%> 

0% 

0% 

0% 

Met  with  or  submitted 
recommendations  to  Medicare 
contractor 

5% 

13% 

8% 

0%. 

0% 

0%, 

Issued  a  Management  Implication 
Report 

5%> 

16% 

19% 

0% 

0% 

4% 

Other 

0% 

7% 

3% 

0% 

0% 

0% 

No  Follow-up  Actions  Taken 

68% 

52% 

41% 

85% 

100% 

93% 

No  Answer 

0% 

3% 

3% 

8% 

0% 

0% 

Total 

n/a 
N  =  19 

n/a 
N  =  31 

n/a 
N  =  37 

n/a 
N  =  13 

n/a 
N  =  1 

n/a 
N  =  27 

Note;       Respondents  were  encouraged  to  cite  all  follow-up  actions  taken  as  a  result  of  tfie  case,  so  rows  are  not  mutually 
exclusive  and  percentages  sum  to  more  tfian  100%. 

Source:  Abt  Associates,  1997 

•  Exploration  for  providers  exhibiting  the  same  pattern  -  Cases  with  exploration  for  other 
instances  have  more  than  tripled,  from  I  \  9c  pre-ORT  to  38%  post-ORT. 

•  Subsequent  written  policy  recommendations  -  Cases  that  lead  to  an  explicit  policy 
recommendation  increased  from  59c  to  11%. 


•   Meeting  with  someone  outside  OIG  concerning  the  case  -  Investigators  met  with  someone 
outside  the  agency  in  1 1  %  of  pre-ORT  cases  and  27%  of  post-ORT  cases.  In  other  words,  the 


14     Note  that  these  follow-up  calegones  are  not  mutually  exclusive,  so  the  percentages  do  not  add  to  lOOVr. 


Abt  Associates  Inc. 


Appendix  7:  Findings  from  the  Survey  of  Cases 


7-23 


proportion  of  cases  on  which  investigators  met  with  someone  outside  the  agency  to  follow-up 
on  the  case  more  than  doubled. 

•  Meeting  with  a  Medicare  contractor  or  submission  of  a  written  recommendation  to  change 
some  aspect  of  contractor  operations  -  Follow-up  work  with  Medicare  contractors  increased 
slightly,  from  5%  to  8%. 

•  Issuance  of  a  Management  Implication  Report  (MIR)  -  MIRs  were  issued  in  5%  of  the  pre- 
ORT  cases,  but  19%  of  the  post-ORT  cases. 

•  No  follow-up  -  Cases  with  no  follow-up  action  declined  substantially,  from  68%  pre-ORT  to 
417c  post-ORT. 

These  results  are  consistent  and  surprisingly  large  for  almost  all  measures.  Meanwhile,  as  shown  on  the 
exhibit,  the  results  for  the  bridge  cases  are  typically  in  between  the  results  for  pre-ORT  and  post-ORT 
cases. 

Exhibit  7-15  offers  some  insight  into  the  timing  of  the  follow-up  that  occurred.  ORT  was  supposed  to 
encourage  earlier  follow-up  by  investigators,  so  that  the  policy  process  would  not  have  to  wait  for  the 
completion  of  a  multi-year  case  in  order  to  learn  from  the  case  (e.g.,  to  take  steps  to  prevent  further 
abuses).  The  exhibit  shows  the  timing  of  follow-up.  In  35%  of  the  post-ORT  cases  -  but  only  26%;  of  the 
pre-ORT  cases  -  follow-up  occurred  at  what  might  be  termed  an  "early"  moment:  before  the  case  closed 
and/or  before  the  case  was  referred.  This  result  is  interesting  because  the  post-ORT  figure  is  likely  an 
understatement,  since  the  post-ORT  cases  are  younger  on  average  and  thus  less  completely  developed  in 
terms  of  follow-up. 


Exhibit  7-15 

Survey  of  Cases:  Timing  of  Follow-up 

Non-Exclusion 

Exclusion 

Follow-up  actions  were  taken... 

Pre 

Bridge 

Post 

Pre 

Bridge 

Post 

...before  the  case  was  referred 

5% 

3% 

5% 

8% 

0% 

0% 

...before  the  case  closed 

21% 

26% 

24% 

0% 

0% 

7% 

...before  the  case  closed  and  before 
the  case  was  referred 

0% 

3% 

5% 

0% 

0% 

0% 

No  Answer 

5% 

13% 

24% 

8% 

0% 

4% 

Not  Applicable/No  Followup  Actions 
Taken 

68% 

55% 

41% 

85% 

100% 

89% 

Total 

100% 
N  =  19 

100% 
N  =  31 

100% 
N  =  37 

100% 
N  =  3 

100% 
N  =  1 

100% 
N  =  27 

Note:       Percentages  may  not  sum  to  100%  due  to  rounding 

Source:  Abt  Associates,  1997 

9.        The  hypothetical:  how  would  the  case  have  been  different  if  it  occurred  in  a 
different  period?  (Question  13) 
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The  final  case-specific  question  we  asked  was  an  attempt  in  a  very  approximate  way  to  use  each  case  as 
its  own  control.  For  pre-ORT  cases,  we  asked  how  the  case  would  have  been  different  under  ORT.  For 
post-ORT  cases,  we  asked  how  the  case  would  have  been  different  before  ORT.  The  question  was  open- 
ended  -  answers  were  not  prompted  by  check  boxes  or  other  means.  The  results  were  as  follows  (Exhibit 
7-16):" 


Exhibit  7-16 

Survey  of  Cases:  Hypothetical:  How  would  case  have  been  different 
if  conducted  during  a  different  period? 

Non-Exclusion  Exclusion 


Difference 

Pre 

Post 

Pre 

Post 

Collaboration  in  some  form 

6% 

11% 

0% 

0% 

Travel  funds 

6% 

6% 

0% 

0% 

Staff  resources 

19% 

11% 

0% 

0% 

Other  resources 

6% 

11% 

8% 

4% 

Change  in  priorities 

0% 

9% 

8%. 

0% 

Other  differences 

0% 

3% 

0% 

0% 

No  difference 

63% 

34%> 

77% 

73% 

Don't  know 

6% 

14% 

0% 

0% 

No  Answer 

0% 

14% 

15% 

23%. 

Totar 

n/a 
N  =  16 

n/a 
N  =  35 

n/a 
N  =  13 

n/a 
N  =  26 

Note:       Respondents  were  given  open-ended  opportunity  to  respond,  so  some  mentioned  multiple 
differences.  As  a  result,  rows  are  not  mutually  exclusive  and  percentages  do  not  sum  to 
100%. 


1)  Responses  to  tfiis  question  for  2  ORT  non-exclusion  cases  and  10RT  exclusion  case  were 
excluded  due  to  a  statement  that  ttie  case  was  a  pre-ORT  case  -  a  statement  that  makes  it 
impossible  to  determine  the  meaning  of  answers  about  how  the  case  then  differed  from  a  pre- 
ORT  case.  3  pre-ORT  case  survey  responses  to  this  question  were  excluded  for  similar 
reasons. 

Source:  Abt  Associates,  1997 

•  Pre-ORT  cases  -  In  over  half  the  cases,  respondents  did  not  indicate  any  respect  in  which  the 
pre-ORT  case  would  have  been  different  under  ORT.  For  the  balance  of  the  pre-ORT  cases, 
the  principal  differences  cited  are  resources  (especially  staff,  as  well  as  other)  and 
collaboration. 

•  Post-ORT  cases  -  Again  in  over  half  the  cases,  respondents  did  not  indicate  any  respect  in 
which  the  post-ORT  case  would  have  been  different  under  the  old,  pre-ORT  regime.  The  most 


15     Note  thai  no  difference  was  inferred  when  respondenls  gave  the  following  survey  answers;  some  form  of  "ihere  would  t>e  no  change." 
some  form  of  '  don't  know."  and  no  answer  Note  also  that  we  do  not  present  results  for  the  bndge  cases  on  this  question,  since  the 
bndge  cases  have  no  obvious  "opposite  "  in  time  -  bndge  cases  are  both  pre-  and  post-ORT  in  various  respects,  so  a  query  asking  about 
the  pre-ORT  or  post-ORT  period  makes  no  obvious  sense. 
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frequent  instances  noted  are  collaboration,  resources  (travel,  staff,  and  other),  and  the  change 
in  management  priorities  in  support  of  the  case. 

From  this  point  of  view,  the  big  difference  ORT  made  is  in  collaboration  and  resources.  That  does  not 
come  as  a  big  surprise,  since  those  are  probably  the  two  most  salient  characteristics  of  ORT,  after  all.'" 

7.6    Case-specific  Survey  Results:  Exclusion  Cases 

As  noted  earlier,  exclusion  cases  are  so  different  from  non-exclusion  cases  that  they  must  be  analyzed 
separately.  These  cases  are  generally  simpler  in  their  characteristics,  and  the  effects  of  ORT  on  them  were 
likely  to  be  minimal.  Accordingly,  our  review  of  these  results  will  be  brief.  Note  that  the  exhibits  are  all 
presented  above. 

1.  Changes  in  the  initiation  of  cases  (Questions  1  and  4) 

Over  90%  of  the  exclusions,  before  and  after  ORT,  were  referred  by  MFCUs  (Exhibit  7-4).  There  was 
some  increase  in  ancillary  collaboration  for  case  identification  after  ORT,  from  64%  to  78%  (Exhibit  7-5). 

2.  Changes  in  the  complexity  of  cases  (Question  2) 

None  of  the  pre-ORT  or  post-ORT  exclusion  cases  was  viewed  as  complex,  in  comparison  to  typical  pre- 
ORT  cases  (Exhibit  7-6). 

3.  Changes  in  other  unusual  case  features  (Question  3) 

There  were  no  unusual  features  ascribed  to  any  of  the  pre-ORT,  bridge,  or  post-ORT  exclusion  cases 
(Exhibit  7-7). 

4.  Case  development  1:  Collaboration  (Question  5) 

There  was  essentially  no  change  in  collaboration  for  case  development  after  ORT.  Of  all  exclusion  cases, 
69%  -  70%  of  cases  had  no  collaboration  (pre-  and  post-ORT),  while  30%  had  at  least  one  collaboration 
(Exhibit  7-8a). 

5.  Case  development  2:  Resources  (Questions  6,  7,  and  8) 

This  is  one  of  the  few  areas  in  which  there  were  some  pre/post  differences  for  exclusion  cases.  None  of 
the  pre-ORT  exclusion  cases  had  unusual  resource  requirements  (Exhibit  7-9).  That  was  also  true  for  just 


16  ll  could  be  argued  lhai  the  results  to  this  query  are  surprising  for  what  they  do  not  discuss:  the  more  frequent  positive  ouicoines  and  the 
far  inore  pro-active  follow-up  after  ORT.  suggested  in  answers  to  our  quenes  about  ca,se  outcomes  and  follow-up  (Questions  10  -  12) 
There  are  many  possible  explanations  why  respondents  did  not  note  that.  say.  for  a  pre-ORT  case,  the  difference  under  ORT  would  be  a 
more  likely  positive  outcome  and  more  active  follow-up.  For  example,  the  respondents  may  simply  have  been  focused  on  inputs  rather 
than  outcomes;  their  references  to  collaboration  and  resources  (in  various  forms)  may  actually  have  been  a  reference  to  these  positive 
outcomes  and  pro-active  approach;  or  the  positive  outcomes  and  pro-active  approach  may  be  evident  in  aggregated  statistics  but  not  yet 
in  direct  observation  We  do  not  have  evidence  to  resolve  this  admittedly  speculative  problem  in  any  definitive  way 
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over  three-quarters  of  the  post-ORT  cases.  But  there  were  a  few  post-ORT  exclusion  cases  for  which 
unusual  resources  were  required,  notably  staff  resources. 

Meanwhile,  there  is  only  a  slight  indication  of  post-ORT  exclusion  cases  for  which  resources  cases  limited 
development  (11%,  versus  0%  for  pre-ORT  cases  -  see  Exhibit  7-11).  Neither  pre-ORT  nor  post-ORT 
cases  made  any  use  of  overnight  travel  (Exhibit  7-10). 

6.  Case  development  3:  Unusual  or  notably  sophisticated  investigative  techniques, 
compared  to  the  typical  pre-ORT  case  (Question  9) 

None  of  the  exclusion  cases,  pre-ORT  or  post-ORT,  involved  sophisticated  investigative  techniques 
(Exhibit  7-12). 

7.  Case  outcomes  (Question  10) 

Virtually  all  pre-  and  post-ORT  exclusion  cases  involve  exclusion  as  a  case  outcome  (no  surprise  there). 
What  is  interesting  is  that  there  is  some  indication  of  a  higher  rate  of  formal  criminal  charges  being 
brought  in  exclusion  cases,  post-ORT  (19%,  versus  8%  pre-ORT  -  see  Exhibit  7-13). 

8.  Case  follow-up  (Questions  1 1  and  12) 

There  is  little  change  in  case  follow-up  results,  pre-ORT  to  post-ORT  (Exhibits  7-14  and  7-15). 

9.  The  hypothetical:  how  would  the  case  have  been  different  if  it  occurred  in  a 
different  period?  (Question  13) 

Respondents  saw  no  hypothetical  change  in  almost  any  of  the  pre-  and  post-ORT  cases  (Exhibit  7-16). 

7.7    Investigator-specific  Results 

A  final  way  in  which  we  sought  to  use  investigators'  opinions  to  understand  ORT  was  to  piggy-back 
investigator-specific  questions  on  our  case-specific  survey.  Specifically,  in  order  to  gain  a  better 
understanding  of  investigators'  overall  opinions  of  ORT,  we  asked  the  investigators  to  look  beyond  the 
case  at  hand  and  to  elaborate  on  their  views  of  ORT.  We  had  interviewed  01  investigators  during  site 
visits  but  wanted  to  confirm,  through  a  confidential  questionnaire,  their  opinions. 

7.7.1     Nature  of  the  Questions 

As  noted  in  Exhibit  7-3  above,  we  posed  four  questions  to  investigators:  what  has  worked  well  in  ORT, 
what  has  not.  and  what  parts  of  ORT  should  and  should  not  be  adopted.  Answers  were  un-prompted  (there 
were  no  check  boxes).  Ample  space  was  provided  so  that  investigators  could  respond  freely.  Their  open- 
ended  responses  were  then  coded  into  a  pre-developed  coding  scheme.  These  subjective  appraisals  of  ORT 
give  us  some  insight  into  how  field  personnel  view  ORT. 
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7.7.2  Sample 


While  we  received  1 28  usable  case  surveys,  the  questions  in  this  section  are  investigator-specific,  not  case- 
specific.  Thus,  survey  instructions  directed  investigators  to  answer  the  overall  opinion  questions  only  once, 
so  as  not  to  duplicate  answers.  Where  individual  investigators  inadvertently  completed  this  section  more 
than  once  (e.g.,  when  the  questions  were  answered  on  more  than  one  survey  by  what  clearly  was  one 
investigator),  responses  were  entered  only  once.  Whjle  our  instructions  were  specific  and  our  review  of 
responses  careful,  we  cannot  be  sure  that  we  avoided  all  duplicates.  For  that  reason,  the  results  of  the 
investigator-specific  questions  should  be  viewed  as  approximations. 

The  N  for  each  question  is  based  on  the  number  of  different  investigators  that  responded  to  that  question, 
not  the  number  of  surveys  received.  Exhibit  7-17  summarizes  the  N  by  question. 


Exhibit  7-17 

Survey  of  Investigators:  Number  Who  Responded  to  Opinion  Questions 

Question  N 

17.  What  in  ORT  has  worked  well?  50 

1 8.  What  in  ORT  has  not  worked  well?  37 
19a.  What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt?  35 
19b.  What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopt?  21 
Source:  Abt  Associates,  1997 


7.7.3    Investigators'  Responses 

What  in  ORT  has  worked  well?  (Question  14) 

As  shown  in  Exhibit  7-18,  nearly  half  of  the  50  investigators  who  expressed  their  opinion  believe  that 
resources  have  been  the  most  helpful  aspect  of  ORT.  The  general  category  of  "All  Other  Resources" 
included  such  things  as:  vehicles,  equipment,  offices,  and  funds  in  general.  Agents  claim  that  more 
funding  has  allowed  them  to  conduct  witness  interviews,  request  OAS  assistance,  and  obtain  medical 
review. 

Collaboration  was  reported  by  36%  of  agents  as  a  notable  improvement.  Agencies  specifically  mentioned 
by  investigators  included:  OAS.  FBI,  HCFA,  FIs,  US  Postal  Inspection  Service,  and  OEI.  On  a  more 
general  note,  one  investigator  pointed  to  interactions  between  federal,  state,  and  local  sectors  of  the 
govemment.  Another  agent  appreciated  DoJ's  increased  dedication  to  enforcement  efforts  against  health 
care  abuse. 

A  second  type  of  resource  -  staff  -  also  rated  highly.  26%  of  agents  feel  that  manpower  increases  permit 
them  to  pursue  more  cases  and  complete  cases  in  a  more  timely  fashion.  According  to  one  investigator, 
the  increase  in  OI  staff  under  ORT  enabled  some  OI  agents  to  specialize.  More  staff  resources  allow 
agents  to  specialize,  which  in  turn  enhances  productivity  on  provider-specific  cases. 
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Exhibit  7-18.  Survey  of  Investigators: 
What  in  ORT  Has  Worked  Well? 

60% 

i 

I 


50%  46% 


All  Other  Resources  Collaboration  Staff  Resources  Travel  Funds  Change  In  Pnorities  Don't  Know  Other 


Response  Category 

Note  N  =  50  I 
Source  Abt  Associates.  19971 
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A  third  type  of  resource  -  travel  -  was  noted  by  \6%  of  the  investigators.  These  investigators  suggested 
that  the  expanded  travel  budgets  allowed  01  to  do  more  cross-region  cases  and  more  complex 
investigations  (e.g.,  cases  that  involved  a  single  provider  with  offices  in  more  than  one  location). 

The  ORT-induced  change  in  priorities  was  perceived  as  important  by  \27c  of  respondent  agents.  Along 
with  agent  specialization,  ORT's  change  in  case  focus  from  general  program  vulnerabilities  to  specific 
aberrant  providers  was  indicated  as  a  significant,  worthwhile  change. 

What  in  ORT  has  not  worked  well?  (Question  15) 

More  than  half  of  the  37  agents  who  answered  found  no  fault  with  ORT  (i.e.,  they  reported  that  "nothing" 
(N=5)  has  not  worked  well  or  that  they  are  unaware  (N=14)  of  what  has  not  worked  well).  Through  the 
eyes  of  field  personnel,  most  aspects  have  ORT  succeeded.  As  depicted  in  Exhibit  7-19,  a  few 
investigators  mention  ORT  and  non-ORT  partnerships  (with  such  agencies  as  FBI,  MFCUs,  HCFA.  and 
other  state  offices)  that  did  not  happen  as  smoothly  as  hoped.  While  interagency  involvement  on  specific 
cases  has  worked,  one  agent  opined  that  collaboration  on  broader  issues,  for  example  trends  and  schemes, 
needs  to  be  addressed. 


Exhibit  7-19 

Survey  of  Investigators:  What  in  ORT  has  not  worked  well? 


Response  Category 

N 

%  of  Investigators 
Who  Responded 

Unknown 

14 

38% 

ORT  Process/Structure  Restrictive 

6 

16% 

Nothing 

5 

14% 

ORT  Partner  Collaboration 

4 

11% 

Outside  Collaboration 

3 

8% 

Excessive  ORT  Administration 

3 

8% 

Other 

3 

8% 

Resources  Still  Too  Limited 

1 

3% 

Intrinsic  Structural  Problems 

1 

3% 

Total  Investigators  who  responded:  37 


Source:  Abt  Associates,  1997 


Some  agents  (8%)  commented  that  too  much  emphasis  was  placed  on  meetings,  updates,  and  conferences, 
which  took  time  away  from  the  investigative  process.  On  the  other  hand,  meetings  and  conferences 
facilitate  the  communication  and  collaboration  processes  that  have  been  vital  to  ORT's  success  and  that 
most  agents  claim  to  have  found  helpful,  so  this  opinion  appears  to  be  a  minority  view. 
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What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt?  (Question  16a) 

While  50  investigators  elaborated  on  what  has  worked  well  in  ORT.  only  about  two-thirds  of  agents 
(N=35)  provided  comments  on  aspects  of  ORT  that  were  worthy  of  being  adopted  by  non-ORT  regions. 
After  setting  aside  the  seven  agents  who  "didn't  know",  the  proponion  drops  to  50%  of  the  total 
investigators  responding.  The  manner  in  which  these  questions  are  phrased  could  lead  investigators  to 
think  that  ORT  aspects  that  worked  well  are  also  worthy  of  adoption  and  therefore  did  not  respond  to  this 
question,  or  it  could  imply  that  just  because  there  are  things  that  worked  well  in  ORT  does  not  mean  they 
should  be  employed  by  non-ORT  regions.  Exhibit  7-20  details  investigators'  recommendations  of  the  parts 
of  ORT  that  should  be  implemented  in  other  regions. 

Of  the  28  agents  who  provided  notable  comments  to  this  question,  almost  40%  feel  that  intra-  and  inter- 
agency collaboration  initiatives  should  be  exercised  by  non-ORT  regions.  Joint  work  with  the  FBI  has 
expedited  cases,  and  cooperation  with  OAS  on  civil  and  criminal  cases  has  also  facilitated  OI's  work. 

The  next  major  set  of  responses  came  in  the  "Other"  response  category  and  was  comprised  of 
miscellaneous  suggestions  such  the  desirability  of  ORT-style  training,  investments  in  agent  specialization, 
education  on  DHHS  programs,  community  outreach,  and  publicity  in  targeted  industries. 

A  small  proportion  (8%)  of  agents  felt  that  ORT's  change  in  focusing  on  certain  providers,  specifically 
DME  suppliers,  HHAs,  and  nursing  homes,  should  be  adopted  in  other  regions.  One  elaborated  that  the 
targeted  provider  types  should  include  hospitals. 

Surprisingly  only  two  agents  commented  on  staff  resources  and  even  though  16%  of  agents  thought  that 
travel  funds  were  one  of  ORT's  most  important  contributions,  none  of  the  investigators  singled  out  travel 
resources  as  an  urgent  issue  for  non-ORT  regions. 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopt?  (Question  16b) 

Nearly  75%  of  the  21  investigators  who  responded  to  this  question  are  unaware  of  anything  that  should 
not  be  adopted  by  non-ORT  regions.  This  implies  that  most  agents  believe  that,  on  the  whole.  ORT 
brought  positive  changes  to  OI  and  that  most  aspects  of  ORT  are  worthy  of  being  adopted  by  other  regions. 

As  depicted  in  the  table  presented  in  Exhibit  7-21,  minimal  complaints  do  exist,  however.  Among  the 
miscellaneous  comments,  agents  explained  that:  staff  should  not  be  hired  and  trained  for  only  ORT-type 
cases  nor  should  they  be  limited  only  to  01  functions;  to  some  extent,  a  "separatism  mentality"  exists;  and 
case  types  should  not  be  limited  in  focus. 

Summary  of  Investigators'  Comments 

The  survey  of  investigators  suggests  that  agents  are  basically  satisfied  with  ORT.  While  some  agents  were 
irritated  by  certain  aspects  of  ORT  (e.g.,  the  frequent  meetings),  each  such  complaint  seemed  more  than 
offset  by  a  larger  set  of  agents  who  found  the  associated  objectives  of  the  irritant  (e.g..  collaboration)  to 
be  one  of  ORT's  strengths. 
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Exhibit  7-20.  Survey  of  Investigators: 
What  single  part  of  ORT  should  non-ORT  regions  adopt? 


60%  - 

I 


I 


I 

t 


40% 


Response  Category 


Nole  N  =  35 

Source  Abl  Associates.  1997 
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Exhibit  7-21 

Survey  of  Investigators:  What  single  part  of  ORT  should  nof  be  adopted? 
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1 

5% 

Other 
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Total  Investigators  who  responded:  21 


Source:  Abt  Associates,  1997 


It  is  generally  fair  to  say  that  the  agent-specific  survey  mirrors  the  case-specific  survey.  From  the  agents' 
points  of  view,  the  most  consistent  strengths  of  ORT  are  collaboration  and  resources.  This  view  contrasts 
somewhat  with  the  view  of  auditors  (see  Appendix  8).  Auditors  were  more  inclined  to  see  ORT  as 
prompting  procedural  innovation  -  a  change  in  focus  -  than  were  agents.  In  this  difference  between  agents 
and  auditors,  we  see  a  fairly  concise  reflection  of  what  our  case  studies  have  suggested  (see  Appendix  1 ), 
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Attachments 


L 


Abt  Associates  Inc. 


May  1,  1997 

Dear  01  Investigator: 

In  April  1995,  the  Department  of  Health  and  Human  Services,  along  with  several  other  federal  and 
state  agencies,  launched  Operation  Restore  Trust  (ORT).  Abt  Associates  is  under  contract  to  the 
Health  Care  Financing  Administration  (HCFA)  to  evaluate  the  ORT  demonstration.  In  order  to 
explore  how  ORT  has  affected  the  identification,  development  and  outcomes  of  01  cases,  we  are 
surveying  investigatorsaboutpre-ORT  cases  (i.e.,  cases  closed  before  April  1995)  and  ORT  period 
cases  (i.e.,  cases  closed  after  April  1995). 

To  date,  we  have  interviewed  01  investigators  about  ORT,  but  have  not  collected  systematic 
qualitative  information  about  a  sample  of  specific  cases.  The  enclosed  survey  is  designed  for  that 
purpose.  We  would  like  you  to  complete  this  survey  in  order  to  help  us  answer  key  questions  about 
some  of  the  major  accomplishmentsof  ORT  such  as:  Are  cases  differently  targeted  under  ORT?  Are 
particular  resources  especially  helpful?  Has  ORT  changed  the  timing  and  level  of  collaboration 
among  key  participants?  With  your  help,  we  will  be  able  to  determine  whether  the  answers  to  these 
questions  differ  among  pre-  and  ORT  period  cases. 

Enclosed  please  find  the  Survey  of  a  Sample  of  OI  Cases:  Interview  Questions.  We  ask  that  you 
complete  the  questionnaire  within  the  ne.xt  week.  In  some  instances,  particular  investigators  will 
have  a  large  number  of  surveys  to  complete.  In  that  event,  we  understand  that  a  slight  delay  may 
occur  in  completing  surveys  for  all  of  the  cases.  Please  return  surveys  directly  to  us  using  the 
enclosed  pre-addressed,postage-paidenvelope.  If  you  have  any  questions,  please  feel  free  to  contaa 
Ms.  Selene  Angier,  Survey  Coordinator,  at  (800)  265-9591  or  via  Internet  E-mail: 
selene_angier@abtassoc.com. 

We  appreciate  your  time  and  effort  in  filling  out  this  very  important  survey. 


Robert  F.  Coulam,  Ph.D. 
Project  Director 
Abt  Associates 

cc:  James  Kopf,  Chief  Inspector,  01 


55  Wheeler  Street  ■  Cambridge.  .Massachusetts  USA  ■  02138-1168  ■  617  492-: relephone  •  6\'  492-52  \'i faisw/ile 


Sincerely, 


Notes  to  Survey  Hypothetical 
Q  13.  -  Survey  of  Cases 


In  order  to  prevent  confusion,  we  designed  two  instruments —  a  pre-ORT  version  and  a  post-ORT 
version  —  for  the  Survey  of  Cases. 

The  following  version  of  the  Survey  of  Cases  is  designed  for  pre-ORT  cases  (i.e.,  audits  that  closed 
before  ORT  began  on  April  I,  1995).  In  the  post-ORT  version,  the  Hypothetical  question  is  worded  as 
follows: 

This  case  was  open  at  least  some  time  after  ORT  began  on  April  1.  1995.  How  would  this  case 
have  been  different  if  it  had  been  completed  before  ORT? 


SURVEY  OF  A  SAMPLE  OF  OI  CASES: 
INTERVIEW  QUESTIONS 


Abt  Associates  is  under  contract  with  the  Heahh  Care  Financing  Administration  (HCFA)  to  carry  out  the 
Evaluation  of  Operation  Restore  Trust.  As  part  of  our  evahiation,  we  are  sun  eying  01  investigators  in 
order  to  better  understand  the  initiation,  development,  and  outcomes  of  pre-  and  post-ORT  cases.  Please 
be  aware  that  any  information  obtained  through  this  survey  is  confidential.  Your  identity  will  not  be  made 
known  to  anyone  outside  of  Abt  Associates,  in  any  published  reports  or  in  any  other  form.  Your  completed 
questionnaires  are  to  be  sent  directly  to  Abt  Associates.  In  addition  to  answering  the  questions  posed  below, 
please  verify  and,  if  necessary,  correct  the  following  information: 

Case  ID#: 
State/Region: 
Subject  Type: 
Date  Case  Opened: 
Date  Case  Closed: 
Period: 


Your  role  in  the  investigation: 


Case  target  or  subject: 

1.  What  prompted  the  investigation?  How  were  the  subject(s)  of  the  investigation  identified? 
[Check  one  only.] 

□  Referral  from  OAS 

□  Referral  from  OI  in  another  region 

□  Referral  from  HCFA 

□  Referral  from  HCFA  contractor 

□  Referral  from  MFCU  or  other  state  agency 

□  I  don't  know 

□  Other —  Fvplain-   

2.  By  comparison  to  cases  before  ORT  (i.e.,  before  April  1,  1995),  was  the  initiation  of  this  case 
unusual  or  atypical  in  any  of  the  following  ways?  /Check  all  thai  apply. I 

□  No        Yes-  Explain:    □   Number  of  targeted  providers?   

□  Location  of  targeted  providers?  

□  Complexity  of  alleged  scheme  or  suspected  infraction?  


□  Complexity  of  investigation?. 

□  Other  

□  Not  applicable,  this  was  a  pre-ORT  case 

3.   Was  there  anything  else  unusual  about  this  case? 

□  No  □  Yes-  Specify:  
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Collaboration: 


4.   With  respect  to  the  identification  of  the  case  subject(s),  did  you  have  any  notable  collaboration 
with;  ICheck  all  thai  apply.} 


Federal  Government 

□  OAS? 
OEI? 
OEC? 
OLC? 
HCFA? 
AoA? 
FBI? 

Other  federal  contractor  or  agency?  Specify": 
State  Government 

□  Long-term  care  ombudsmen? 
Medicaid  agency? 
MFCUs? 

State  survey  or  certification  staff? 
Prosecutors? 

Other  state  contractor  or  agency?  Specify:— 
Qui  tam? 
Hot  Line? 
Other?  Specify 


□ 
□ 
□ 
□ 
□ 
□ 
□ 


□ 
□ 
□ 
□ 
□ 


□  01  in  another  region? 

□  U.S.  Attorney's  office? 

□  Medicare  contractors? 

□  Peer  review  organization? 


□ 
□ 
□ 
□ 


No  notable  collaboration  with  respect  to  identifying  subject(s) 


Briefly  describe  the  nature  of  the  collaboration,  if  any: 


5.    With  respect  to  the  development  of  the  case  itself,  did  you  have  any  notable  collaboration  with: 
[Check  all  that  apply.] 


Federal  Government 

□  OAS? 
OEI? 
OEC? 
OLC? 
HCFA? 
AoA? 
FBI? 

Other  federal  contractor  or  agency?  Specify 
State  Government 

□  Long-term  care  ombudsmen? 
Medicaid  agency? 
MFCUs? 

State  survey  or  certification  staff? 
Prosecutors? 

Other  state  contractor  or  agency?  Specify:— 


□ 
□ 
□ 
□ 
□ 
□ 
□ 


□ 
□ 

□ 
□ 
□ 


□  01  in  another  region? 

□  U.S.  Attorney's  office? 

□  Medicare  contractors? 

□  Peer  review  organization? 


□  Other?  Specify:  

□  No  notable  collaboration  with  respect  to  development  of  the  case 
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Briefly  describe  thp  nature     'V^^  collaboration,  if  any: 


Resources: 

6.  Relative  to  the  typical  01  investigation,  did  case  development  require  an  unusual  amount  of: 
[Check  all  that  apply.] 

□  Staff  time? 

□  Travel? 

□  Computer  resources? 

□  Medical  review? 

□  Communication  costs? 

□  Legal  resources? 

□  Other?  

□  None  of  the  above 

7.  Was  case  development  limited  by  the  shortage  of  any  particular  type  of  resource? 

□  No  □  Yes-Specify:  


8.  Approximately  how  much  overnight  travel  did  this  case  require? 

□  None  □   Some  (e.g.,  less  than  20  days)        □   A  lot  (e.g.,  20  days  or  more) 
Methods: 

9.  Relative  to  the  typical  01  case  conducted /)e/oreORT  (i.e.,  before  April  1.  1995).  were  any  unusual 
or  notably  sophisticated  investigative  techniques  used  in  this  case? 

□  No,  nothing  unusual  □   Yes —  Specify:  


□  Not  applicable,  this  was  a  pre-ORT  case 
Results  (Insofar  as  you  know  them): 

10.  With  respect  to  this  case,  have  any  of  the  follow  ing  outcomes  occurred?  [Check  all  that  apply.] 

□  Referral  to  U.S.  Attorney's  office  or  state  prosecutor 

□  Referral  to  FBI 

□  Referral  to  OEC 

□  Referral  to  OLC 

□  Formal  criminal  charges 

□  Formal  civil  charges 

□  Exclusion 

□  Recoupment 

□  Other —  Specify:  

□  None  of  these  outcomes  has  occurred 

□  Don't  know 


1 1 .  Did  you  take  any  of  the  follow-up  actions  listed  below  as  a  result  of  the  c^se?  [Check  all  that  apply. I 

□  Commenced  investigations  of  similar  provider(s) 

□  Submitted  written  recommendations  for  policy  changes  (e.g.,  changes  in  federal  statutes  or 
regulations)  Specify  


□  Met  with  an  agency  outside  OIG.  Which  agenc(ies)?  

□  Met  with  a  Medicare  contractor  or  submitted  written  recommendations  to  change  some  aspect 
of  contractor  operations 

□  Issued  a  Management  Implication  Report  (MIR) 

□  Other   .  


□  No  followup  actions  taken 

12.  If  you  took  any  of  the  above  followup  actions,  did  you  do  so  before; 

□  the  case  was  referred  to  another  law  enforcement  agency  (e.g.  FBI,  DOJ,  or  state  prosecutor)? 

□  the  case  closed? 

□  Not  applicable,  no  followup  actions  taken 
Hypothetical: 

13.  This  case  closed  before  ORT  began  on  April  1,  1995.  How  would  this  case  have  been  different 
under  ORT? 


Overall  Opinions  on  ORT: 

Senior  OIG  managers  are  particularly  interested  in  your  views  on  how  ORT  has  generally  affected  your 
work.  For  that  purpose,  the  questions  below  look  beyond  this  one  case  and  ask  you  to  compare  the 
ORT  and  pre-ORT  periods  generally.  (If  you  are  completing  surveys  for  more  than  one  case  and  have 
already  answered  these  questions  on  another  survey  regarding  a  different  case,  you  do  not  have  to 
repeat  your  answers  here.  But  please  be  sure  to  answer  the  questions  at  least  once.) 

14.  What  in  ORT  has  worked  well? 
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15.  What  in  ORT  has  not  worked  well? 


16.  If  you  were  addressing  a  group  of  your  colleagues  from  non-ORT  regions,  what  single  part  of  ORT 
would  you  urge  them: 

a.   To  adopt? 


b.     Not  to  adopt? 
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Appendix  8 
Findings  from 
The  Survey  of  Audits 

8.0  Introduction 

As  discussed  in  the  preceding  appendix,  the  evaluation  of  Operation  Restore  Trust  (ORT)  must  use 
multiple  evaluation  approaches  to  establish  reliable  understandings  of  what  ORT  has  and  has  not 
accomplished.  The  survey  of  audits  is  a  further  response  to  that  requirement. 

The  need  for  this  survey  follows  from  the  same  logic  as  our  survey  of  cases.  Like  cases,  audits  by  the 
Office  of  Audit  Services  (OAS)  are  a  key  window  through  which  to  view  a  wide  and  diverse  spectrum  of 
ORT  effects.  Certain  major  effects  ORT  sought  to  achieve  -  e.g.,  to  enhance  collaboration  -  were 
designed  to  register  on  the  identification  and  development  of  audits.'  We  therefore  have  special  reason 
to  focus  on  the  character  and  frequency  of  ORT  effects  on  these  basic  building  blocks  of  enforcement. 

Audits  are  tracked  in  an  automated  system,  the  Audit  Information  Management  System  (AIMS).  AIMS 
tracks  OAS  audits  from  the  date  the  audit  is  opened  until  the  date  the  final  audit  report  is  issued.  The 
system  details  audit  sources,  subject  types  and  locations,  and  information  on  the  development  of  audits 
through  the  audit  flow  cycle.  While  this  data  source  is  the  foundation  for  quantitative  analyses  by  Abt 
Associates  (see  Appendix  5),  it  does  not  provide  enough  qualitative  information  for  us  to  analyze  how 
ORT  has  affected  certain  key  characteristics  of  audits.  Specifically,  AIMS  does  not  provide  information 
on  critical  changes  suggested  in  our  interviews,  such  as: 

•  changes  in  the  purpose  of  audits, 

•  the  initiation  of  promising  new  forms  of  collaboration  (e.g.,  with  state  surveyors), 

•  the  establishment  of  new  methods  of  targeting  audits, 

•  the  availability  under  ORT  of  key  resources  (e.g.,  travel)  to  permit  more  far-reaching  audits, 
and 

•  other  possible  effects. 

To  fill  in  this  information,  we  needed  to  collect  a  few  qualitative  details  about  a  large  number  of  audits. 
To  that  end,  we  surveyed  lead  auditors  involved  with  each  audit  in  a  random  sample  of  audits.  As  outlined 
below,  our  original  sample  was  the  universe  of  audits  of  ORT-type  providers  that  have  opened  since  1993. 


1      Of  course,  audits  are  not  the  only  object  of  ORT  effons  For  example.  ORT  was  also  designed  to  affect  the  identification  and 

development  of  cases  by  the  Office  of  Investigations  in  OIG  (as  noted  in  Appendix  7),  as  well  as  to  affect  the  operations  of  the  Agency 
on  Aging  and  of  HCFA  and  its  contractors.  Only  some  of  the  effects  on  these  other  agencies  would  yield  new  audits.  But  ORT  funds 
were  devoted  in  large  pan  to  audit  (and  case)  development  by  the  OIG.  and  changes  in  audit  processes  were  a  principal  target  of  the 
ORT  approach  Thus,  while  they  are  not  the  only  target  of  ORT,  audits  were  a  particularly  important  object  of  ORT. 
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The  survey  permitted  us  to  collect  information  not  recorded  in  AIMS  and  enables  us  to  answer  key 
questions  about  some  of  the  major  accomplishments  of  ORT. 

We  have  asked  a  few  specific  questions  of  each  audit  (see  survey  instrument  in  Attachment  1  to  this 
appendix).  As  with  the  survey  of  cases,  the  survey  of  audits  was  not  a  review  of  audit  records,  nor  a 
complete  history  of  each  sampled  audit.  But  it  did  identify  selected  characteristics  of  the  sampled  audits. 
The  key  premise  of  our  analysis  is  that,  by  aggregating  this  information  across  all  sampled  audits,  we  can 
quantify  the  prevalence  of  certain  audit  characteristics  that  ORT  sought  to  affect. 

We  had  hoped  to  use  the  survey  to  estimate  pre/post  differences  in  audit  characteristics.  However,  there 
were  very  few  pre-ORT,  ORT-type  audits  in  the  ORT  states  (note  our  summary  of  these  audits  in  our 
AIMS  analysis  in  Appendix  6).  As  it  turns  out,  attrition  among  lead  auditors  for  those  few  pre-ORT  audits 
was  relatively  large,  so  that  knowledgeable  auditors  were  not  available  for  most  of  the  few  pre-ORT 
audits.  As  a  result,  our  pre-ORT  sample  for  audits  is  trivially  small  -  two  audits.  This  number  of  audits 
is  too  small  even  for  bare  descriptive  purposes  of  pre-ORT  audits,  never  mind  a  pre/post  analysis.  We  will 
ignore  those  two  pre-ORT  audits  in  the  analysis  that  follows.  Fortunately,  what  we  lack  in  a  pre-ORT 
sample  is  made  up  by  a  correspondingly  large  number  of  post-ORT  responses  (104).  This  post-ORT 
sample  will  permit  us  to  undertake  a  detailed  descriptive  analysis  of  audits  under  ORT. 

The  survey  of  audits  was  designed,  implemented,  and  analysed  by  Abt  Associates,  as  described  in  the 
report  that  follows.  However,we  obviously  could  not  survey  auditors  in  the  Office  of  the  Inspector 
General  without  significant  assistance  from  OIG's  Office  of  Audit  Services.  We  appreciate  the  assistance 
OAS  gave  us  in  this  effort;  we  especially  would  like  to  thank  the  auditors  who  took  the  time  to  complete 
the  surveys.  Of  course,  Abt  Associates  is  entirely  responsible  for  the  interpretation  of  results  that  follows. 

8.1    Summar7  of  Findings  and  Organization  of  the  Report 

The  message  from  our  survey  of  audits  is  fairly  clear  and  consistent:  for  OAS,  ORT  represents  a  change 
in: 

•  approach,  toward  more  provider-specific  audits, 

•  process,  toward  more  collaboration, 

•  techniques,  toward  more  data-driven  methods,  and 

•  resources,  toward  more  needed  funding,  particularly  in  travel. 
This  summary  is  fairly  congruent  with  what  ORT  tried  to  accomplish. 

It  is  interesting  to  note  the  contrast  of  these  audit  results  to  the  results  for  the  survey  of  cases  (Appendix 
7).  For  investigators,  the  big  difference  ORT  made  was  in  collaboration  and  resources.  01  investigators 
saw  ORT  as  a  ramping  up  of  slightly  changed  enforcement  methods.  By  contrast,  OAS  auditors  see  ORT 
as  a  more  fundamental  change  in  technique  and  approach.  That  result  comports  more  or  less  exactly  with 
the  findings  of  our  process  analyses  (Appendix  1 ). 
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Our  discussion  of  the  survey  of  audits  is  organized  as  follows: 

8.2  Design  and  methodology,  including  a  discussion  of  the  descriptive  design  we  used  in  our 
survey  and  how  we  assigned  audits  to  pre-  and  post-ORT  periods. 

8.3  Data  Collection,  including  descriptions  of  our  sample  design,  our  survey  instrument,  and  our 
procedures  for  conducting  the  survey  (particulady  those  concerning  confidentiality). 

8.4  Presentation  of  responses  to  our  audit-specific  questions. 

8.5  Presentation  of  responses  to  our  auditor-specific  questions. 

8.2    Design  and  Methodology 

We  will  analyze  the  survey  of  audits  using  a  simple  descriptive  model.  In  this  section,  we  explain  that 
choice  and  certain  decisions  that  influence  how  it  should  be  applied. 

8.2.1     The  Impracticality  of  a  Pre/Post  or  Demonstration/Comparison  Model 

As  we  have  argued  elsewhere  (see  preceding  appendix;  and  Abt  Associates,  1996a),  the  ideal  model  for 
estimating  demonstration  effects  is  a  so-called  "double-difference"  model.  We  were  unable  to  pursue  this 
model  in  our  analysis  of  audits  for  two  reasons: 

•  The  sample  sizes  would  have  to  be  large  enough  to  permit  multivariate  estimation  of  the 
model's  parameters;  even  with  100%  response  rates  on  the  universe  of  audits,  there  were  not 
enough  pre-ORT,  ORT-area  audits  in  the  demonstration  states  to  do  so  (there  were  only  10 
such  audits). 

•  Leading  auditors  responsible  for  most  of  the  pre-ORT  audits  in  the  ORT  states  had  left  their 
regions  or  retired,  so  that  the  pre-ORT  sample  declined.  Our  original  sample  included  all  of 
the  ORT-area,  pre-ORT  audits  done  in  the  ORT  states.  As  noted  in  Section  8.3  below,  we 
lost  8  of  those  audits  due  to  the  departure  of  the  responsible  lead  auditors.  As  a  result,  our 
pre-ORT  responses  total  two,  too  few  to  be  a  reliable  baseline. 

Three  options  were  available  to  rescue  the  pre/post  design,  but  none  was  satisfactory.  First,  we  could 
expand  the  universe  of  audits  to  include  non-ORT-subject-area  audits,  of  which  there  are  many  more  in 
the  pre-ORT  period.  Unfortunately,  audits  of  non-ORT  auditee  types,  such  as  hospitals  and  laboratories, 
tend  to  be  so  different  from  audits  of  ORT-type  providers  that  comparisons  of  the  characteristics  of  the 
two  would  be  essentially  meaningless.  Second,  we  could  expand  the  pre-ORT  period  (i.e.,  include  audits 
that  opened  before  1993).  But  there  were  not  many  ORT-type  audits  in  earlier  years  either;  and  problems 
of  recall  error  and  staff  turnover  would  only  increase  as  we  went  further  back  in  time.  Finally,  we  could 
expand  the  universe  to  include  comparison  state,  pre-ORT,  ORT-subject-area  audits.  But  quite  apart  from 
the  conceptual  problems  of  using  non-ORT  regions  to  establish  a  pre-ORT  baseline  for  ORT  regions,  there 
were  few  of  these  audits  in  the  non-ORT  states  in  the  pre-ORT  period.  As  a  result,  even  if  including  these 
other  audits  were  defensible  on  technical  grounds,  there  were  not  enough  of  them  to  solve  the  problem. 
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In  the  end,  we  could  not  rescue  the  pre/post  design  for  the  simple  reason  that  ORT  was  such  a  major 
change  in  OAS  audit  targets  that,  as  it  turned  out,  no  defensible  pre-ORT  baseline  could  be  estimated.  The 
big  change  in  ORT  was  the  fact  of  performing  this  type  of  provider-specific,  ORT-subject-area  audit,  not 
simply  the  way  the  audit  was  performed. 

Given  the  small  number  of  pre-ORT  audits  in  our  sample,  we  were  able  to  create  a  particularly  large 
sample  of  ORT-period  audits:  1 14.  This  sample  size  gives  us  a  considerable  body  of  detail  on  all  the 
different  kinds  of  ORT-period  audits.  It  also  permits  us  to  stratify  the  results  -  e.g.,  by  region  and  subject 
type  -  in  a  way  that  a  smaller  post-ORT  sample  (e.g.,  as  in  the  survey  of  cases)  would  not  permit.  We 
have  exploited  this  difference  in  our  analysis  of  results.  We  are  thus  able  to  offer  a  description  of  ORT 
audits  that  is  somewhat  more  complex  than  would  otherwise  be  possible. 

8.2.2  Assigning  Audits  to  "Pre-ORT"  and  "Post-ORT"  Periods 

All  of  the  above  discussion  of  course  assumes  that  we  can  assign  audits  to  pre-  and  post-ORT  periods. 
Unfortunately,  the  date  field  included  in  the  AIMS  data  we  received  only  permits  us  to  identify  the  year 
that  audits  opened.  Our  decision  rule  was  simple.  We  assigned  audits  that  opened  in  1993  and  1994  to 
the  pre-ORT  period  and  audits  that  opened  in  1995  and  1996  to  the  post-ORT  period.  This  specification 
makes  the  "pre-ORT"  period  somewhat  overinclusive,  if  anything.  The  reason;  since  some  of  the  audits 
opening  in  1994  doubtless  extended  into  the  ORT  period,  it  is  likely  that  -  unbeknownst  to  us  -  some  of 
the  audits  we  are  identifying  as  pre-ORT  might  more  properly  be  considered  "bridge  audits"  (using  logic 
analogous  to  that  set  forth  for  "bridge  cases"in  Appendix  7).  Our  counts  of  pre-ORT  audits  will  tend  to 
be  too  large,  so  the  problems  set  forth  above  -  too  few  pre-ORT  audits  in  areas  of  interest  to  us  -  are  not 
created  by  a  definition  of  pre-ORT  that  is  too  restrictive.  If  anything,  the  reverse  is  true. 

8.2.3  Principal  Limitations  of  the  Analysis 

Our  survey  of  audits  provides  far  more  discriminating  information  on  certain  audit  characteristics  than  is 
available  from  any  other  source.  It  is  worth  noting  the  limitations  of  this  data,  however: 

•  Our  data  do  not  cover  a  long  enough  period  to  capture  all  the  audit  effects  that  will  ultimately 
occur  -  We  have  noted  repeatedly  that  the  brevity  of  our  evaluation  requires  us  to  score 
demonstration  efforts  before  all  of  the  effects  of  demonstration  activities  have  occurred  or 
been  recorded.  In  the  survey  of  audits,  this  means  that  where  the  passage  of  time  is  important 
to  a  characteristic  -  e.g.,  to  aspects  of  audit  development,  outcomes,  and  follow-up  -  answers 
to  our  survey  questions  will  be  based  on  necessarily  incomplete  information  and  will  tend  to 
understate  the  ultimate  presence  of  the  characteristics. 

•  Some  of  potentially  the  most  effective  audits  -  those  that  are  at  particularly  sensitive  stages 
-  have  been  deleted  from  the  sample  -  We  had  to  delete  7  audits  from  our  sample  due  to 
various  kinds  of  confidentiality  requirements.  It  seems  reasonable  to  assume  that  audits  in 
these  sensitive  stages  are  among  the  more  productive  audits.  While  we  are  not  doing  a 
pre/post  analysis  of  audits,  the  deletion  of  these  audits  will  tend  to  bias  downward  our 
estimates  of  the  presence  of  certain  audit  characteristics  -  e.g.,  the  proportion  of  audits  with 
some  kind  of  active  follow-up. 
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•  The  survey  measures  frequency,  not  intensity  -  What  we  said  in  Appendix  7  about  our  survey 
of  cases  is  also  true  of  our  survey  of  audits:  our  survey  is  better  at  capturing  some  effects  on 
a  lot  of  audits  than  large  effects  on  a  few  audits  -  or,  more  generally,  at  capturing  notable 
changes  in  frequency  than  notable  changes  in  intensity. 

In  view  of  these  limitations,  we  need  to  be  careful  in  how  we  interpret  the  survey  data.  The  important 
point  is  that  -  even  in  view  of  these  intrinsic  limitations  -  the  survey  provides  an  unusually  rich  picture 
of  audits  in  the  ORT  period. 


8.3    Data  Collection 


8.3.1     Sample  Design 

As  noted  in  the  preceding  section,  our  sample  of  ORT-state,  ORT-subject  area  audits  is  actually  the 
universe  of  such  audits.  Audits  that  opened  before  1 993  were  excluded  due  to  concerns  about  recall  error 
and  the  effects  of  staff  movement  and  turnover  at  OAS. 


AIMS  contains  150  audits  of  ORT-type  providers  that  opened  up  during  or  after  1993.  Of  the  150  ORT- 
type  audits,  140  took  place  during  ORT  (i.e.,  opened  during  or  after  1995)  and  10  were  conducted  before 
ORT.  Prior  to  the  pilot  test,  OAS  requested  that  7  audits  (2  pre-ORT  and  5  ORT)  be  deleted  from  the 
sample,  as  these  audits  were  at  sensitive  stages  that  precluded  their  being  discussed  with  parties  outside 
formal  proceedings.  Exhibit  8-1  shows  the  audit  sample  distribution  by  period,  state,  and  provider  type. 


Exhibit  8-1 

Survey  of  Audits:  Sample  Distribution  of  Usable  ORT-period  Surveys  by  State  and  Subject  Type 

Subject  Type 


State 

DME 

HHA 

SNF 

Other 

Total 

California 

3 

6 

10 

2 

21 

Florida 

1 

10 

21 

3 

35 

Illinois 

2 

1 

3 

4 

10 

New  York 

7 

12 

6 

6 

31 

Texas 

1 

2 

2 

2 

7 

Total  Audits  in  Sample 

14 

31 

42 

17 

104 

Note:  2  usable  pre-ORT  surveys  were  returned:  a  Florida  HHA  and  a  Texas  HHA. 
Source:  Abt  Associates,  1997 


Exhibit  8-2  shows  the  number  and  distribution  of  the  audits  we  were  ultimately  able  to  use  in  our  analysis. 
The  response  rate  was  extremely  high  (138  out  of  143  surveys  sent,  or  almost  97%),  thanks  largely  to 
active  follow-up  efforts  by  OAS.  The  proportion  of  usable  surveys  (after  deletions  for  canceled  audits  and 
other  factors)  was  74%  of  the  total  sent. 


Abt  Associates  Inc. 


Appendix  8:  Findings  from  the  Survey  of  Audits 


8-5 


Exhibit  8-2 


Survey  of  Audits:  Number  of  Surveys  Sent  and  Received  by  Period 

Pre-ORT 

ORT 

Total 

Total  Audit  Surveys  Sent  [net  of  7  audits 
deleted] 

8 

135 

143 

Not  Received 

0 

5 

5 

Received 

8 

130 

138 

Not  Usable' 

6 

26 

32 

Usable  (N) 

2 

104 

106 

Usable  Surveys  as  %  of  Surveys 
Sent 

25% 

77% 

74% 

Reasons  include;  canceled  audit,  related  case  pending  with  a  grand  jury,  CIN  is  an  administrative  account,  limited  OAS 
involvement,  pre-audit  account 


Source:  Abt  Associates,  1 997 

8.3.2  Survey  Instrument 

A  copy  of  the  survey  instrument  and  cover  letters  are  included  in  Attachment  1  to  this  appendix.  The 
questionnaire  was  revised  on  several  occasions —  notably,  after  feedback  from  OAS  in  January  1997,  and 
after  the  pilot  test  in  April  1997.  The  survey  form  in  Attachment  1  is  the  version  that  was  distributed  to 
auditors  when  we  fielded  the  full  survey. 

The  survey  asks  for  a  series  of  qualitative  answers  on  certain  key  areas  of  interest  in  light  of  the  ORT 
intervention.  The  survey  questions  in  paraphrased  form  are  as  shown  on  Exhibit  8-3.  As  the  exhibit 
suggests,  we  used  the  opportunity  of  this  audit-specific  survey  to  ask  some  auditor-specific  questions. 

To  minimize  burden,  we  limited  the  survey  to  5  pages.  Most  questions  were  close-ended  (i.e.,  required  the 
respondent  only  to  check  a  box).  Where  we  were  unable  to  propose  check-box  answers,  we  provided 
space  for  very  brief,  open-ended  responses. 

8.3.3  Pilot  Test 

After  receiving  feedback  from  OAS  on  our  initial  proposal  for  the  audit  survey,  we  revised  the  pilot 
version  of  the  survey  instrument  and  accompanying  letters.  In  April  1997,  ten  pilot  test  survey  packets 
were  mailed  directly  to  the  associated  lead  auditors.  Based  on  the  responses  we  received  on  those  test 
surveys,  we  revised  the  survey,  by  developing  new  check-box  responses  for  several  questions.  These 
changes  were  designed  principally  to  make  it  easier  for  auditors  to  complete  the  surveys  and  to  reduce 
possible  ambiguities  in  responses  to  certain  questions. 

8.3.4  Conduct  of  the  Survey 

After  receiving  further  feedback  from  OAS,  we  established  the  following  procedure  for  the  survey.  At  the 
beginning  of  May  1997  we  launched  our  field  effort.  Auditors  for  each  audit  received: 
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Exhibit  8-3 

Survey  of  Audits:  Summary  of  Survey  Questions 

Question  Type 

and  Number 

Question 

AUDIT-SPECIFIC  QUESTIONS 

1 

Wliat  was  the  primary  purpose  of  the  audit  (e.g..  to  identify  questionable  charges,  target 

aberrant  providers,  etc.)? 

2 

What  prompted  the  audit? 

3 

Were  any  special  analyses,  data  runs,  or  other  efforts  undertaken  to  identify  subjects? 

4 

Was  the  initiation  of  this  audit  unusual  or  atypical  (e.g.,  in  terms  of  the  number  of  providers 

targeted,  their  location,  the  complexity  of  the  scheme,  or  the  complexity  of  the  investigation 

itself)? 

5 

Was  there  anything  else  unusual  about  this  audit? 

6 

With  respect  to  identification  of  audit  subjects,  did  you  have  any  notable  collaboration  with 

federal,  state,  and  other  agencies? 

7 

With  respect  to  development  of  the  audit,  did  you  have  any  notable  collaboration  with 

federal,  state,  and  other  agencies? 

8 

Did  audit  development  require  any  unusual  amount  of  resources  (e.g.,  staff,  travel,  etc.)? 

9 

Was  audit  development  limited  by  the  shortage  of  any  particular  type  of  resource? 

10 

Approximately  how  much  overnight  travel  did  this  audit  require? 

1 1 

Relative  to  the  typical  pre-ORT  audit,  was  there  anything  unusual  about  how  this  audit  was 

conducted? 

12 

Is  the  audit  complete?  If  not,  is  a  final  report  likely  to  be  issued? 

13 

Have  certain  important  outcomes  occurred  (e.g.,  referral  to  other  agencies  (e.g.,  01), 

recoupment,  formal  charges,  etc.)? 

14 

Were  any  follow-up  actions  taken  as  a  result  of  the  audit  (e.g.,  written  policy 

recommendations,  exploration  to  find  others  exhibiting  a  similar  pattern  of  misbehavior, 

etc.)? 

15 

Did  the  audit  result  in  the  identification  of  questioned  costs?  How  much? 

16 

How  would  this  audit  have  been  different  under  ORT?  or 

How  would  this  audit  have  been  different  before  ORT?^ 

INVESTIGATOR-SPECIFIC  QUESTIONS 

17 

What  in  ORT  has  worked  well? 

18 

What  in  ORT  has  not  worked  well? 

19a. 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt? 

19b 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopf 

Source:  Abt  Associates  Survey  of  Audits,  1997. 

•      a  letter  of  introduction  from  OAS, 

•      a  letter  of  explanation  from  Abt  Associates, 

•      a  questionnaire  with  pre-printed  audit  information  (Case  Identification  Number,  state/region. 

subject  type,  year  opened,  status,  period  and  audit  manager),  and 

2       Two  versions  of  ihe  questionnaire,  pre  and  post,  were  created  to  facilitate  asking  two  different  versions  of  this  hypothetical  question. 
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•  a  pre-addressed,  postage-paid  return  envelope. 

We  then  tracked  the  survey  responses  over  the  weeks  that  followed.  Weekly  tracking  reports  were  faxed 
to  OAS,  which  followed  up  on  surveys  that  had  not  yet  been  returned.  Surveys  were  received  up  to  June 
30,  1997. 

Because  our  audit  sample  is  the  universe  of  ORT-type  audits  found  in  ALMS,  no  substitute  surveys  were 
distributed  to  replace  attrition.  If  the  lead  auditor  associated  with  an  audit  was  unavailable,  or  if  the  audit 
otherwise  could  not  be  surveyed,  it  was  deleted  from  the  sample.  (The  net  results  are  shown  on  Exhibit 
8-2  above.) 

A  Microsoft  Access  data  entry  system  was  developed  for  the  purposes  of  entering  survey  data.  Using  a 
coding  map,  open-ended  survey  responses  for  the  hypothetical  and  auditors'  overall  opinions  on  ORT  were 
coded  accordingly,  and  then  entered  into  the  database. 

8.3.5  Confidentiality 

The  confidentiality  of  the  survey  of  audits  was  a  major  concern  to  OAS  and  to  individual  auditors.  For 
open  audits  in  general  -  and  for  Ol-assist  audits  in  particular  -  it  was  important  to  ensure  that  our  data 
collection  and  report  did  not  create  any  problems  for  OAS's  work.  To  that  end,  our  audit  survey  report: 

•  Does  not  include  multiple  fields  of  any  audit-specific  survey  responses,  as  might  allow  a  reader 
to  stitch  together  a  series  of  features  to  infer  the  identity  of  an  audit  or  group  of  audits. 

•  Uses  aggregates  of  audits  for  its  analyses,  rather  than  identifying  any  specific  aspect  of  any 
specific  audit. 

•  Does  not  quote  any  audit-specific  open-ended  answers. 

•  Was  reviewed  by  OAS  Central  Office  before  being  circulated  to  others,  to  catch  any  possible, 
inadvertent  disclosures. 

These  procedures  were  designed  to  ensure  that  no  audit  identities  were  revealed  in  our  report.  Even  with 
these  assurances,  several  surveys  were  returned  incomplete,  with  notes  attached  indicating  that  the  audits 
concerned  sensitive  matters  that  precluded  the  auditor  from  completing  the  survey.  These  audits  were 
eliminated  from  our  sample. 

8.4     Audit-specific  Survey  Results 

The  following  section  presents  our  audit-specific  survey  results.  For  convenience,  we  proceed  more  or 
less  consecutively  through  the  survey  questions,  but  have  grouped  clusters  of  questions  together  when  they 
address  the  same  or  similar  issues.  Where  the  results  by  state  or  subject  type"*  are  notably  interesting,  they 
are  presented. 


The  five  possible  subject  types  were  home  health,  durable  medical  equipment  (DME).  hospice,  skilled  nursing  facilities  (SNFs),  and 
other. 
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1.    The  purpose  of  the  audit  (Question  1 ) 


Among  other  things,  ORT  represented  an  effort  to  reorient  the  purpose  of  OAS  audits,  from  "'program" 
audits  to  provider  audits.  The  difference  between  the  two  types  of  audits  is  not  just  one  of  semantics,  as 
the  two  types  of  audits  are  structured  very  differently.  This  difference  is  perhaps  best  understood  in  terms 
of  the  samples  that  are  drawn  for  the  two  types  of  audits.  In  a  program  audit,  the  sample  drawn  involves 
at  most  a  few  claims  from  each  of  many  providers,  thus  permitting  auditors  to  draw  inferences  about  the 
prevalence  of  conditions  or  behaviors  in  the  program.  By  contrast,  a  provider  audit  involves  multiple 
claims  from  a  single  or  few  providers,  thus  permitting  auditors  to  draw  inferences  about  the  prevalence 
of  practices  by  particular  providers.  This  latter  capacity  is  of  obvious  relevance  to  enforcement  efforts.'' 
ORT  opted  for  doing  more  of  the  latter  -  and  then  using  findings  of  questioned  charges  against  a  few 
providers  (rather  than  program-level  audits  as  such)  to  support  discussions  of  program  vulnerabilities.  In 
that  respect,  ORT  represented  a  more  inductive  approach  to  audit  targeting  and  methods. 

In  addition,  of  course,  ORT  represented  a  change  in  the  provider  types  being  audited,  toward  greater 
emphasis  on  the  ORT  targets:  home  health  agencies,  DME  providers,  hospices,  and  skilled  nursing 
facilities. 

Post-ORT  audits  reflect  the  apparent  influence  of  these  objectives: 

•  Type  of  provider  -  Obviously,  as  noted  in  our  discussion  of  ORT  sampling  issues  (see  above), 
the  audits  in  the  ORT  period  focused  on  types  of  providers  that  were  rarely  audited  in  pre-ORT 
period.  There  were  140  ORT-type  (subject  area/state)  audits  in  the  post-ORT  universe,  but  only 
1 0  ORT-type  audits  in  the  pre-ORT  universe. 

•  Audit  purpose  -  As  shown  on  Exhibit  8-4,  virtually  all  of  the  ORT-type  audits  were  provider- 
specific:  only  5%  had  the  primary  purpose  of  appraising  the  "policies,  practices,  and 
vulnerabilities  of  the  Medicare  program."  Most  of  the  audits  surveyed  were  designed  either  to 
identify  aberrant  providers  (34%  of  all  post-ORT  audits)  or  to  identify  questionable  charges 
from  specific  providers  (56%).  A  few  of  the  audits  (5%)  were  designed  to  assist  an  active 
investigation.  This  latter  figure  is  almost  surely  an  understatement,  since  audits  of  this  type  were 
especially  vulnerable  to  being  excluded  from  our  analysis.  Note  that  these  results  do  not  vary 
much  by  state:  ORT  audits  in  all  five  states  are  almost  entirely  provider  specific  (data  not 
shown). 

These  results  suggest  that  ORT  did  indeed  represent  a  different  audit  approach  for  OAS  auditors.  We  will 
expand  on  that  result  below. 


4      li  should  perhaps  be  emphasized  that  neither  of  these  two  approaches  is  necessarily  superior  to  the  other.  <i  priori.  The  appropriate  mix 
of  effons  depends  on  a  large  complex  of  variables,  concerning  the  character  of  provider  infractions,  the  effectiveness  of  other 
enforcement  processes,  and  so  on  The  important  point  for  understanding  ORT  is  that  (among  other  things)  ORT  embodied  an 
assumption  that  more  provider-specific  audits  needed  to  be  done. 
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Exhibit  8-4 

Survey  of  Auditors:  Primary  Purpose  of  ORT  Audits 

Response 

%  of  Post-ORT  Audits 

Identify  or  target  aberrant  providers  for  audit  and/or  0!  referral 

56% 

IHpntifv  niiPQtinn^ihlp  ph^irnP^  frnm  Qnpf^ifip  nrnv/iHpr^Q^  fnr  rprrMinmpnt 

and  otfier  follow-up  actions 

34% 

Assist  an  active  investigation 

5% 

Appraise  policies,  practices,  and  vulnerabilities  of  thie  f\/ledicare  program 

5% 

other 

1% 

No  answer 

0% 

Total 

100% 
N  =  104 

^       Total  may  not  sum  to  100%  due  to  rounding 

Source:  Abt  Associates,  1997 

2.    Initiation  of  the  audit  1:  Wtiat  prompted  ttie  audit  and  the  use  of  collaboration  in 
identifying  audit  subjects  (Questions  2  and  6) 

ORT  was  designed  -  perhaps  more  than  anything  else  -  to  be  a  partnership  among  agencies.  One  way  we 
might  see  the  effects  of  this  would  be  in  relatively  high  proportions  of  referrals  from  other  agencies,  such 
as  HCFA  and  01.  At  the  same  time,  however,  high  rates  of  self-referral  might  indicate  the  successful 
effects  of  another  ORT  goal:  to  encourage  data-driven  efforts  to  target  providers. 

Our  results  are  shown  on  Exhibit  8-5a.  There  are  two  messages  in  the  exhibit: 

Exhibit  8-5a 

Survey  of  Audits:  Audit  Target  or  Subject 

Response                                                                    Percent  of  Post-ORT  Audits 

Self-initiated  by  OAS 

58% 

Referral  from  HCFA 

23% 

Referral  from  HCFA  contractor 

6% 

Referral  from  OAS  in  another  region 

3% 

Request  from  01 

3%> 

Referral  from  f^FCU  or  other  state  agency 

1% 

Other 

7% 

Total 

100% 
N  =104 

Note:  Percentages  may  not  sum  to  100%  due  to  rounding. 

Source:  Abt  Associates,  1997 
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•  Self-refenral  -  Most  audits  (587f )'  are  self-referred.  This  high  percentage  makes  later  questions 
in  the  survey  (e.g.,  about  collaboration  in  identifying  subjects  and  data-driven  methods) 
commensurately  more  interesting. 

•  HCFA  -  The  percentage  of  HCFA  referrals  (23%)  was  surprisingly  high  to  us,  although  these 
are  almost  all  Florida  audits  (see  below).  An  additional  69c  of  the  ORT  audits  were  referred  by 
Medicare  contractors. 

•  01  request  -  The  percentage  of  Ol-request  audits  (3%)  was  surprisingly  low  to  us.  Site  visits 
to  auditors  in  the  field  have  emphasized  the  growing  importance  of  Ol-request  audits;  indeed, 
one  of  the  major  conclusions  of  our  case  studies  is  that  ORT  was  important  for  bringing  OAS 
and  01  closer  together,  at  a  more  timely  point  in  enforcement  actions  (see  Appendix  1 ).  But  our 
survey  data  do  not  suggest  a  very  large  quantity  of  such  efforts.  One  possible  explanation:  a 
disproportionate  share  of  Ol-request  audits  were  at  sensitive  stages,  so  Ol-request  audits  were 
disproportionately  deleted  from  our  sample. 

•  Results  by  state  -  The  results  by  state  show  somewhat  more  interesting  variation.  As  shown  on 
Exhibit  8-5b,  three  states  had  somewhat  unusual  audit  referral  patterns.  First,  and  most 
interesting,  63%  of  the  Florida  ORT  audits  were  referred  by  HCFA,  far  more  than  for  any  other 
ORT  state  (three  ORT  states  had  0%  ,  while  California  had  10%:).  Most  of  these  are  SNF  audits 
(data  not  shown).  Second,  California  has  a  much  higher  rate  of  contractor  referrals  than  any 
other  state  (24%,  compared  to  3%,  0%,  0%,  and  0%  for  the  other  four  states).  This  result  may 
be  an  indication  of  the  effectiveness  of  Medicare  contractors  in  California  in  partnering  with 
California's  ORT  steering  committee.^  Most  of  these  California  audits  are  home  health  audits 
(data  not  shown).  Finally,  Illinois  has  a  relatively  high  rate  of '"extramural  OAS  referrals"  -  i.e., 
referrals  from  other  OAS  offices.  The  figure  is  0%  for  the  other  four  ORT  states,  but  30%  for 
Illinois.  However,  the  N  is  so  low  on  these  Illinois  audits  (3)  that  not  much  should  be  read  into 
this  particular  result. 

Returning  to  the  combined  data,  we  also  used  a  second  approach  to  asking  about  collaboration  in  the 
initiation  of  the  audit.  In  Question  6,  we  provided  a  long  list  of  agencies  and  asked  if  there  had  been  any 
notable  collaboration  with  each  of  these  agencies  in  the  identification  of  audit  subjects.  The  results  are 
shown  on  Exhibit  8-6.  That  exhibit  shows  high  proportions  (50%  -  60%)  of  the  audits  involved  at  least 
some  collaboration  with  HCFA  and  Medicare  contractors  to  identify  audit  subjects.  (More  will  be  said 
on  this  result  below.)  Over  23%  of  all  audits  involved  some  collaboration  with  01.  What  is  particulariy 


5  The  percentage  increases  to  6 1     if  OAS  referrals  from  outside  the  region  are  included. 

6  For  a  review  of  one  contractor's  work  with  the  California  ORT  committee,  see  the  discussion  of  TransAmenca  in  Appendix  3. 
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Exhibit  8-5b 

Survey  of  Audits:  Audit  Target 

or  Subject  by  State 

Percent  of  Post-ORT  Audits 

Response  California 

Florida 

Illinois 

fill!  1 \y 1 W 

New  York 

Texas 

Total 

Self-initiated  by  OAS 

57% 

29% 

60% 

84% 

86% 

n/a 

Referral  from  HCFA 

10% 

63% 

0% 

0% 

0% 

n/a 

Referral  from  HCFA 
contractor 

24% 

3% 

0% 

0% 

0% 

n/a 

Referral  from  OAS  in 
another  region 

0% 

0% 

30% 

0% 

0% 

n/a 

Request  from  01 

10% 

3% 

0% 

0% 

0% 

n/a 

Referral  from  MFCU  or  other 
state  agency 

0% 

0% 

0% 

3%, 

0% 

n/a 

Other 

0% 

3% 

10% 

13% 

14% 

n/a 

Total 

100% 

100% 

100% 

100% 

100% 

n/a 
N  =  104 

Note:  Columns  may  not  sum  to  100%  due  to  rounding. 

Rows  do  not  sum  to  100%  since  columns  are  not  interdependent. 

Source:  Abt  Associates,  1997 

Exhibit  8-6 

Survey  of  Audits:  Notable  Collaboration  in  the  Identification  of  Audit  Subjects 

Response 

Percent  of  Post-ORT  Audits 

Medicare  contractor 

59% 

HCFA 

54%, 

OIG/OI 

23% 

State  survey/certification  staff 

22% 

OAS  in  another  region 

8% 

State  Medicaid  agency 

8% 

OIG/OEI 

4% 

FBI 

3% 

U.S.  Attorney's  Office 

3% 

Peer  Review  Organizations 

2% 

Other  State  agencies 

2% 

Other  Federal  agencies 

2% 

Agency  on  Aging 

1% 

Long-Term  Care  Ombudsmen 

1% 

None 

17% 

Total 

n/a 

N  =104 

Note:  Respondents  were  encouraged  to  cite  all  aspects  of  notable  collaboration,  so  rows  are  not  mutually  exclusive  and 

percentages  may  not  sum  to  100%. 

Source:  Abt  Associates,  1997 
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interesting  is  that  almost  the  same  proportion  of  all  audits  (22%)  involved  collaboration  with  state  survey 
and  certification  staffs.  This  result  comports  well  with  our  site  visit  information  and  underscores  how 
productive  this  particular  federal-state  linkage  was.  It  is  also  a  result  that  is  fairly  attributed  entirely  to 
ORT,  since  -  by  all  the  evidence  available  to  us  -  the  linkage  of  OIG  auditors  with  state  survey  and 
certification  staffs  is  without  precedent  before  ORT. 

3.    Initiation  of  the  audit  2:  Tlie  use  of  special  analyses  or  data  runs  to  identify  audit 
subjects  (Question  3) 

The  hope  for  ORT-type  audits  was  that  the  targeting  of  the  audits  would  be  more  data-driven  than  in  the 
past.  We  cannot  speak  to  the  change  from  the  past;  but  it  is  clear  from  Exhibit  8-7  that  virtually  all  ORT 
audits  (89%)  involved  special  data-based  efforts  to  identify  subjects.  It  is  worth  noting  that  virtually  all 
ORT  states  (except  California)  and  subject  types  (except  home  health)  have  the  same  response  (over  907c 
"yes")  to  this  question  (data  not  shown). 


Exhibit  8-7 

Survey  of  Audits:  Use  of  Special  Analyses,  Data  Runs,  and  Other  Special  Efforts  to  Identify  Audit 
Subjects 

Response  Percent  of  Post-ORT  Audits 

Special  efforts  undertaken  89%  

No  special  efforts  11%  

Total  100% 
 N  =104  

Source:  Abt  Associates,  1997 

4.    Initiation  of  the  audit  3:  Something  unusual  about  the  initiation  of  the  audit,  compared 
to  pre-ORT  audits  (Question  4) 

This  is  one  of  the  few  questions  in  the  survey  that  asked  auditors  to  compare  the  audit  before  them  to  a 
pre-ORT  audit.  Auditors  found  something  unusual  in  77%  of  the  audits.  As  shown  in  Exhibit  8-8,  the 
most  common  unusual  attribute  was  multiplicity  of  providers  (49%  of  all  audits),  followed  by  location  of 


Exhibit  8-8 

Survey  of  Audits:  Unusual  Initiation  in  Comparison  to  Pre-ORT  Audits? 

Response  Percent  of  Post-ORT  Audits 

Number  of  targeted  providers  unusual  49%  

Location  of  targeted  providers  unusual  13%  

Complexity  of  suspected  scheme,  abuse,  or  infraction  unusual  13%  

Complexity  of  audit  unusual  8%  

Othier  characteristics  unusual   28%  

Nothing  unusual  20%  

No  answer  

Total  n/a 

 N  =104  

Note:  Respondents  were  encouraged  to  cite  all  the  ways  in  which  the  audit  was  unusual,  so  rows  are  not  mutually  exclusive  and 

percentages  sum  to  more  than  100%. 
Source:  Abt  Associates.  1997  — . 
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the  provider  and  complexity  of  the  audit  scheme  (13%  each),  and  last  by  the  complexity  of  the  audit  itself. 
One-quarter  of  the  audits  found  "other"  unusual  characteristics  in  the  audits. 

5.     Anything  else  unusual  about  this  audit.(Question  5) 


Two-thirds  of  respondents  found  something  else  unusual  about  the  audits  in  question  (Exhibit  8-9).  These 
other  unusual  features  ran  across  a  wide  spectrum,  including  the  character  of  the  subject,  the  use  of  state 
survey  and  certification  staff,  and  others. 


Exhibit  8-9 

Survey  of  Audits:  Anything  Else  Unusual  about  This  Audit? 

Response 

Percent  of  Post-ORT  Audits 

Yes 

66% 

No 

26% 

No  answer 

8% 

Total 

100% 

N  =104 

Note:  Total  may  not  sum  to  1 00%  due  to  rounding. 

Source:  Abt  Associates.  1997 

6.    Development  of  the  audit  1:  Collaboration  (Question  7) 


By  comparison  to  what  we  found  for  the  initiation  of  audits,  our  results  for  the  development  of  audits  give 
heightened  importance  to  one  particular  collaborator:  the  Medicare  contractor.  As  shown  in  Exhibit  8- 
10,  the  Medicare  contractor  played  a  collaborative  role  in  audit  development  in  almost  60%  of  the  audits, 
followed  by  HCFA  itself  in  just  over  47%  of  the  audits.  The  most  notable  other  collaborators  were  state 
government  staffs:  survey/certification  staffs  (15%;  of  audits)  and  Medicaid  agency  staff  (13%).  OI  was 
involved  in  1 1  %  of  the  audits  (again,  a  percentage  that  is  likely  artificially  low,  due  to  the  disproportionate 
exclusion  of  Ol-assist  audits).  One  interesting  vestigial  collaborator  is  state  Peer  Review  Organizations 
(PROs),  which  provide  medical  review  expertise  for  audits  (as  they  do  for  cases  -  see  Appendix  7).  In 
site  visits,  auditors  mentioned  that  one  particularly  useful  result  of  ORT  was  the  ready  availability  of  PRO 
medical  reviewers  to  collaborate  with  auditors  in  the  early  stages  and  execution  of  audits.  In  4%  of  the 
audits  surveyed,  state  PRO  staffs  played  a  notable  collaborative  role. 

To  give  some  context  to  these  comments  on  collaboration-in-development  (as  well  as  the  comments  on 
coUaboration-in-initiation  in  Question  6  above),  consider  the  answer  to  Question  16  below:  how  would 
the  audit  have  been  different  before  ORT?  Respondents  answered  that,  in  over  one-third  of  the  audits 
(35%),  the  collaboration  would  not  have  taken  place.  This  figure  provides  a  rough  lower  bound  on  the 
incremental  effects  that  ORT  brought  to  audit  processes. 
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Exhibit  8-10 

Survey  of  Audits:  Notable  Collaboration  in  Audit  Development 

Response 

Percent  of  Post-CRT  Audits 

Medicare  contractor 

58% 

HCFA 

47% 

State  survey/certification  staff 

15% 

State  Medicaid  agency 

13% 

OIG/OI 

11% 

OAS  in  another  region 

10% 

U.S.  Attorney's  Office 

5% 

Peer  Review  Organizations 

4% 

Other  State  agencies 

3% 

Other 

3% 

MFCUs 

1% 

Other  Federal  agencies 

1% 

FBI 

1% 

OIG/OEI 

0% 

Agency  on  Aging/Long  Term  Care  Ombudsmen 

0% 

None 

13% 

No  answer 

3% 

Total 

n/a 

N  =104 

Note:  Respondents  were  encouraged  to  cite  all  aspects  of  notable  collaboration,  so  rows  are  not  mutually  exclusive  and 

percentages  sum  to  more  than  100%. 

Source:  Abt  Associates,  1997 

7.    Development  of  the  audit  2:  Resources  (Questions  8,  9  and  10) 

When  asked  if  the  audits  involved  any  unusual  resource  demands,  there  were  four  principal  resources 
identified  (Exhibit  8-11): 

•  Computer  resources  -  One-half  of  all  ORT-era  audits  were  judged  to  have  required  unusual 
amounts  of  computer  resources.  This  result  is  consistent  with  the  clear  indications  of  data- 
driven  audits  provided  by  other  survey  questions  (e.g.,  see  our  discussion  of  answers  to  Question 
3  [the  use  of  special  analyses  and  data  runs],  and  Questions  6  and  7  [concerning  collaboration 
with  (data  rich)  partners,  such  as  the  Medicare  contractors]).  Its  importance  here  is  that  it 
suggests  a  thrust  in  audits  in  keeping  with  the  ambitions  of  ORT:  ORT  resources  were  designed 
in  part  to  "lower  the  price  of  data,"  so  that  data  would  be  more  consistently  used.  While  we  have 
no  pre-ORT  or  comparison  data  to  calibrate  this  answer,  unusual  computer  resources  on  half  of 
all  audits  suggests  relatively  consistent  use. 
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Exhibit  8-11 

Survey  of  Audits:  Unusual  Resources 

Required  for  Audit  Development 

Response 

Percent  of  Post-ORT  Audits 

Computer  resources 

49% 

Travel  resources 

42% 

Medical  review 

38% 

Staff  time 

35% 

Communication  costs 

2% 

Dthpr 

c.  /o 

None  of  the  above 

26% 

No  answer 

4% 

Total 

n/a 

N  =104 

Note:  Respondents  were  encouraged  to  cite  all  aspects  of  unusual  resource  use,  so  rows  are  not  mutually  exclusive  and 

percentages  may  not  sum  to  1 00%. 

Source:  Abt  Associates,  1997 

\ 

I  •      Travel  -  Auditors  claim  42%  of  all  ORT-era  audits  used  unusual  amounts  of  travel.  How  much 

travel?  Exhibit  8-12  provides  an  indication  (for  all  audits,  not  just  for  the  42%  with  unusual 
amounts  of  travel):  60%  of  all  ORT-era  audits  used  "some"  travel  (defined  in  the  survey  as 
under  20  overnight  days),  while  22%  used  a  "lot"  of  travel  (defined  as  20  or  more  overnight  days 
of  travel).  On  site  visits,  auditors  emphasized  the  value  of  travel  funds.  These  survey  answers 
suggest  how  frequently  it  was  needed. 

It  is  interesting  to  note  the  contrast  to  the  answers  for  post-ORT  cases  : 

-  84%  of  the  cases  required  no  overnight  travel,  compared  to  only  16%  of  the  audits. 


Exhibit  8-12 

Survey  of  Audits:  Amount  of  Overnight  Travel  Required  by  the  Audit 

Response 

Percent  of  Post-ORT  Audits 

A  lot  (defined  as  20  days  or  more) 

22% 

Some  (less  than  20  days) 

60% 

None 

16% 

No  answer 

2% 

Total 

100% 
N  =104 

Note:  Total  may  not  sum  to  100%  due  to  rounding. 

Source:  Abt  Associates,  1997 

-  14%  of  the  cases  required  some  overnight  travel,  compared  to  60%  of  the  audits. 


I 
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-  None  of  the  cases  required  a  lot  of  overnight  travel,  compared  to  227f  of  the  audits. 

It  is  thus  clear  from  this  survey  data  that  the  ORT  partner  that  most  uses  the  added  travel 
resources  of  ORT  is  OAS.^  It  is  worth  noting  that  New  York  and  Texas  had  the  most  audits 
with  some  travel  (both  at  71%  of  their  state  total),  while  Illinois  had  the  least  (20%,  although 
Illinois  did  have  a  relatively  high  proportion  of  audits  with  a  lot  of  travel  -  40%:  data  not 
shown). 

•  Medical  review  -  One  of  the  more  striking  results  shown  on  Exhibit  8-11  is  that  unusual  medical 
review  resources  were  consumed  in  nearly  40%  of  all  audits.  This  prevalence  estimate  echoes 
the  comments  in  site  reviews,  about  the  greater  use  in  audits  of  medical  and  clinical  staffs  from 
PROs,  Medicare  contractors,  HCFA,  and  others.  Under  ORT,  these  staffs  were  said  in 
interviews  to  be  an  integral  part  of  the  planning  and  execution  of  many  audits,  rather  than  as  an 
add-on  or  sequel  to  a  standard  audit.  Our  survey  supports  that  claim. 

•  Staff  time  -  One-third  of  all  ORT  audits  were  said  to  have  used  unusual  amounts  of  staff  time. 
ORT  did  not  give  OAS  nearly  as  many  additional  auditors  as  it  gave  01  investigators:  OAS  got 
2  auditors,  compared  to  the  24  new  investigators  for  01.  But  the  staff  that  were  added  (and  the 
reorientation  of  experienced  staff*)  represented  an  expansion  of  the  staff  pool  for  performing 
these  audits.  It  appears  that  one-third  of  the  audits  benefitted. 

Overall,  these  answers  for  audits  suggest  that  ORT's  liberalization  of  resources  was  important  to  OAS. 
We  cannot  be  very  rigorous  in  calibrating  these  results,  since  we  do  not  know  what  proportion  of  audits 
used  unusual  amounts  of  staff  time  before  or  outside  of  ORT.  But  there  is  in  these  answers  a  more 
consistent  citation  of  unusual  resource  needs  in  audits  than  in  investigations  (see  Appendix  6).  There  is 
also  a  slightly  lesser  claim  of  resource  limits  constraining  enforcement:  where  case  development  was  said 
to  be  limited  by  resource  shortages  in  27%  of  all  ORT  cases,  that  was  true  in  only  22%;  of  all  audits  (data 
not  shown). 

8.    Development  of  the  audit  3:  Unusual  audit  techniques  (Question  1 1) 

The  survey  responses  described  to  this  point  suggest  that  audits  under  ORT  have  had  an  unusual  focus, 
used  unusual  amounts  of  resources,  and  embodied  frequent  use  of  data-driven  methods.  Not  surprisingly, 
then,  in  Question  1 1  auditors  found  that  ORT  audits  were  conducted  in  an  unusual  way  in  80%)  of  the 
audits  (Exhibit  8-13).'  Almost  three-quarters  of  all  audits  were  said  to  have  employed  "an  unusual 
approach";  an  overlapping  one-quarter  were  said  to  have  employed  a  "notably  sophisticated  technique." 
The  15%  who  checked  "other"  noted  a  variety  of  different  things.  While  these  terms  are  of  course 
imprecise,  they  reinforce  the  view  that  is  suggested  by  answers  to  earlier  questions:  ORT  audits  were 


7  One  caution  is  in  order  in  this  conclusion.  It  is  of  course  possible  for  the  lesser  01  travel  in  fact  to  be  more  valuable  at  the  margin,  even 
if  audits  consume  more  travel  on  average. 

8  See  the  discussion  of  the  diversion  of  audit  effort  toward  ORT  audits,  in  Appendix  6. 

9  Consider  the  contrasting  figure  for  cases:  investigators  thought  that  there  were  unusual  investigative  techniques  in  only  1 1  %  of  cases. 
See  answers  to  Question  9  of  the  Survey  of  Cases,  in  Appendix  7. 
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different  in  some  important  ways  from  pre-ORT  audits,  in  that  they  embodied  approaches  and  techniques 
uncommon  before  the  ORT  intervention. 


Exhibit  8-13 

Survey  of  Audits:  Unusual  Conduct  of  Audit, 

Compared  to  Pre-ORT  Audit 

Response 

Percent  of  Post-ORT  Audits 

Notably  sophisticated  technique 

27% 

Unusual  audit  approach 

73% 

Other  unusual  features 

14% 

Nothing  unusual 

17% 

Not  applicable/no  answer 

3% 

Total 

n/a 

N  =104 

Note:  Respondents  were  encouraged  to  cite  all  aspects  of  unusual  audit  conduct,  so  rows  are  not  mutually  exclusive  and 

percentages  sum  to  more  than  1 00%. 

Source:  Abt  Associates,  1997 

State-specific  data  show  an  interesting  set  of  results  on  this  question  (data  not  shown).  Two  results  should 
be  noted  in  particular.  First,  in  Florida,  almost  90%  of  the  audits  were  thought  to  have  used  unusual  audit 
approaches.  Second,  in  New  York,  auditors  thought  that  unusual  techniques  were  employed  in  over  60% 
of  the  audits.  This  latter  result  perhaps  reflects  what  was  suggested  to  us  in  our  site  visits  to  New  York: 
that  OAS  there  had  changed  its  technique  to  employ  more  "short-form"  audit  methods  to  permit  quicker 
identification  of  likely  malefactors. 

9.  Development  of  the  audit  4:  Is  the  audit  is  complete,  and  (if  not)  is  an  audit  report  is 
likely  to  be  issued?  (Question  12) 

The  audits  were  complete  in  43%  of  the  sampled  audits;  26%  of  the  incomplete  audits  were  said  to  be 
likely  to  result  in  an  audit  report  (data  not  shown). 

10.  Audit  outcomes  to  date  (Questions  13  and  15) 

The  outcomes  of  the  ORT  audits  to  date  are  summarized  in  Exhibit  8-14.  Of  the  total,  31%  led  to 
recoupment  actions,  21%  to  referrals  of  some  kind  (e.g.,  to  01),  and  15%  led  to  other  outcomes.'"  Only 
30%  led  to  none  of  the  potential  positive  outcomes  listed  .  With  the  passage  of  time,  that  30%  figure  is 
almost  sure  to  decline  for  audits  as  a  whole,  since:  1 )  57%  of  the  audits  are  not  complete  (see  discussion 
of  Question  1 2  above);  and  2)  some  of  the  more  successful  audits  are  disproportionately  excluded  from 
the  sample,  for  reasons  of  confidentiality." 


10  Note  that  these  items  are  not  mutually  exclusive. 

1 1  In  other  words,  when  the  results  of  these  laner  audits  are  added  back  in,  the  proponion  of  audits  without  a  successful  result  will  tend  to 
decline 


Abt  Associates  Inc. 


Appendix  8:  Findings  from  the  Survey  of  Audits 


8-18 


Exhibit  8-14 

Survey  of  Audits:  Significant  Audit  Outcomes 

Response 

Percent  of  Post-ORT  Audits 

Recoupment 

31% 

Referral  to  OIG/OI 

21% 

Other  outcomes 

1 5% 

Exclusion 

3% 

Formal  Civil  or  Criminal  Charges 

4% 

Referral  to  the  FBI 

2% 

Identification  of  other  providers  to  be  audited   2% 


None  of  these  outcomes  28% 


Don't  know/no  answer 

11% 

Total 

n/a 

N  =104 

Note:  Respondents  were  encouraged  to  cite  all  significant  outcomes,  so  rows  are  not  nnutuaily  exclusive  and  sum  of  percentages 


may  not  equal  100%. 
Source:  Abt  Associates.  1997 


By  state  (data  not  shown),  the  dispersion  in  outcomes  was  fairly  wide.  The  proportion  of  audits  with  some 
kind  of  positive  outcome  ranged  from  81%  in  New  York,  to  roughly  60%  for  three  states  (Florida.  Illinois, 
and  Texas),  and  38%  for  California  (data  not  shown).  By  subject  type  (data  not  shown),  DME  cases  had 
the  highest  rate  of  positive  outcomes  (79%),  followed  by  SNF  cases  (69%),  hospice  cases  (57%),  and 
home  health  and  other  (approximately  50%  each). 

Another  measure  of  outcomes  to  date  is  provided  by  answers  to  Question  1 5,  which  asks  whether  the  audit 
resulted  in  the  identification  of  questioned  costs,  and  how  much.  Exhibit  8-15  summarizes  the  results. 


Exhibit  8-15 


Survey  of  Audits:  Distribution  of  Questioned  Costs  by  Audit 


Number  (Percent) 
of  Audits 

Range  of  identified  Costs 

Total  for  All  Audits 
in  the  Range 

Average  per  Audit 
in  the  Range 

44  (42%) 

S  0 

$  0 

$  0 

26  (25%) 

S  0  to  $250,000 

$    2.6  million 

$  0.1  million 

10  (10%) 

$250,001  to  $500,000 

$    3.5  million 

$  0.3  million 

6  (6%) 

$500,001  to  $1,000,000 

$    5.6  million 

$  0.9  million 

17  (17%) 

$1,000,000  to  $100,000,000 

$  158.6  million 

$  9.3  million 

103  (99%)' 

S  0  to  $100,000,000 

S  170.2  million' 

$  1.7  million 

Note  that  one  audit  witti  particularly  large  questioned  costs  has  been  omitted,  to  avoid  isolating  the  costs  it  has  identified. 


Source:  Abt  Associates  Survey  of  Audits,  1997. 
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Obviously,  even  with  an  outlier  audit  excluded,  60  audits  (SS^c  of  ail  audits)  have  identified  over  S 1 70 
million  in  questioned  costs.  These  amounts  will  increase  as  additional  ORT  audits  mature.  Many  audits 
(42%  of  the  total)  identify  no  questioned  costs,  and  a  similar  proportion  (41  %)  identify  relatively  small 
amounts  of  questioned  costs.  The  balance  of  audits  (a  little  under  one-fifth  of  the  total)  account  for  over 
90%  of  all  identified  questionable  costs. 

The  fact  that  there  is  a  portfolio  of  results  -  some  large,  some  not  so  large;  some  audits  leading  to  referrals 
or  recoupments,  some  not  -  is  hardly  surprising.  It  is  probably  true  that  an  audit  process  that  did  not  have 
a  range  of  results  in  these  various  respects  would  not  be  taking  many  chances  or  exploring  important 
possibilities  (e.g.,  small  probabilities  of  large  problems).  We  are  not  in  a  position  to  judge  the  distribution 
of  results  -  it  would  perhaps  be  more  interesting  to  know  the  time  trend  on  results  such  as  these  (e.g.,  over 
time,  does  OAS  get  better  at  targeting  malefactors,  using  its  more  intensive  data  capabilities?).  But  it  is 
at  least  fair  to  say  that  -  in  the  ORT  subject  areas  and  states  -  these  results  are  almost  entirely  incremental, 
since  OAS  basically  did  not  do  audits  like  these  in  the  past.  It  is  probably  also  true  that  results  such  as 
those  shown  in  Exhibits  8-14  and  8-15  are  the  reason  for  one  of  the  important  findings  of  our  process 
analyses  (see  Appendixes  land  2):  collaborators  as  diverse  as  01,  HCFA,  and  Assistant  U.S.  Attorneys 
suggested  that  one  of  the  larger  surprises  of  ORT  was  the  value  of  working  with  auditors. 

JO.  Audit  follow-up  to  date  (Question  14) 

Another  area  where  ORT  sought  to  improve  on  traditional  practices  was  in  the  follow-up  to  audit  efforts. 
In  the  past,  it  was  thought,  people  working  in  these  enforcement  processes  almost  always  failed  to  follow- 
up  on  the  implications  of  their  work  in  a  proactive  way.  Thus,  discovery  of  one  provider  engaging  in  a 
particular  pattern  of  fraud  did  not  typically  lead  to:  1 )  an  active  search  to  find  other  providers  engaged  in 
the  same  type  of  abuse,  or  2)  preventive  efforts  (e.g.,  contacting  HCFA  or  Medicare  contractors  to  discuss 
the  need  for  policy  changes,  in  light  of  the  discovered  abuse).  In  critical  ways,  the  process  failed  to  learn. 

ORT  involved  at  least  a  rhetorical  commitment  to  changing  that  pattern.  Our  survey  results  do  not  identify 
changes  as  such.  But  we  can  characterize  the  activities  in  individual  audits.  As  shown  on  Exhibit  8-1 6a: 

•  Particular  audits  led  to  subsequent  "audits  of  similar  providers"  in  49%  of  the  audits. 

•  Auditors  met  with  Medicare  contractors  or  made  written  recommendations  to  change  some 
aspect  of  contractor  operations  as  a  follow-up  to  28%  of  the  audits. 

•  Meetings  with  someone  outside  OIG  were  held  in  197f  of  the  audits,  and  written 
recommendations  were  submitted  in  1 8%  of  the  audits. 

•  There  was  no  follow-up  in  21%  of  the  audits. 

These  figures  may  represent  a  lot  or  a  little  of  the  kind  of  follow-up  that  ORT  is  seeking.  It  would  be 
useful  to  know  the  pre-  or  non-ORT  norm  against  which  to  compare  these  results.  It  would  also  be  useful 
to  know  how  these  figures  change  over  time.  But  these  figures  do  seem  substantial,  especially  if  we  allow 
for  the  fact  that:  1 )  over  half  of  the  audits  are  not  yet  complete  (see  above),  2)  our  sample  is  biased  to 


12     Note  that  the  follow-up  categories  are  not  mutually  exclusive,  so  the  percentages  do  not  add  to  100%. 
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Exhibit  8-1 6a 

Survey  of  Audits:  Did  the  Respondent  Take  Any  Follow- 

up  Actions? 

Response 

Percent  of  Post-ORT  Audits 

Commenced  audits  of  similar  providers 

49% 

Met  with  a  Medicare  contractor  or  made  written 

recommpnriatinn^  to  rhannp  *5nmp  a«;nprt  nf  rnntr;^rtnr 

1  ^\j\Jl  1  II  1  Id  lUdllta/l  lO  K\J  VyI  lul           Ovyl  1  )C  ClOL/CLfl  \Jt  K^\JI  ILI  CL\jl\JI 

operations 

28% 

Met  with  an  agency  outside  OIG 

19% 

Submitted  written  recommendations  for  policy  changes 

18% 

Other  follow-up 

12% 

No  follow-up 

21% 

No  answer 

2% 

Total 

n/a 
N=104 

Note:  Respondents  were  encouraged  to  cite  all  follow-up  they  had  taken,  so  rows  are  not  mutually  exclusive  and  percentages  sum 
to  more  than  100%. 

Source:  Abt  Associates,  1997 

exclude  some  particularly  productive  audits,  and  3)  of  necessity,  many  audits  are  "dry  holes"  (we  do  not 
have  an  exact  figure)  and  are  unlikely  to  lead  to  follow-up,  even  in  the  best  of  worlds. 

By  state,  there  are  significant  variations  in  follow-up  frequency.  As  shown  on  Exhibit  8- 16b,  follow-up 
audits  of  similar  providers  were  claimed  in  77%  of  the  Florida  audits,  but  only  14%  of  Texas  audits  and 
20%  of  Illinois  audits.  Follow-up  policy  recommendations  were  submitted  in  only  9%  of  the  Florida 
audits  and  10%  of  the  Illinois  audits,  but  in  the  neighborhood  of  40%  of  all  California  and  Texas  audits. 

Overall,  California  and  Illinois  had  no  follow-up  in  approximately  40%  of  the  audits,  Florida  and  New 
York  each  had  follow-up  in  approximately  15%  of  their  audits,  while  Texas  recorded  follow-up  in  all  of 
its  audits. 

1 1.  The  hypothetical:  how  would  ORT  audits  have  been  different  if  it  they  had  occurred 
before  ORT?  (Question  16} 

The  final  audit-specific  question  we  asked  was  an  attempt  in  a  very  approximate  way  to  use  each  audit  as 
its  own  control.  Specifically,  for  each  post-ORT  audit,  we  asked  how  the  audit  would  have  been  different 
before  ORT.  The  question  was  open-ended  -  answers  were  not  prompted  by  check  boxes  or  other  means. 
The  results  are  shown  on  Exhibit  8-17." 


13     Note  that  no  difference  was  inferred  when  respondents  gave  the  following  survey  answers:  some  form  of  "there  would  be  no  change." 
some  form  of  "don't  know."  and  no  answer. 
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Exhibit  8-1 6b 

Survey  of  Audits:  Audit  Tarq 

3t  or  Subject  by 

State 

Percent  of  Post-ORT  Audits 

Response 

California 

Florida 

Illinois 

rjew  TorK 

Texas 

Total 

Commenced  audits  of  similar 
providers 

38% 

77% 

d\J  /o 

A  OO/ 

4<;  /o 

14% 

n/a 

Met  with  a  Medicare  contractor 
or  made  written 
recommendations  to  change 
some  aspect  of  contractor 
operations 

48% 

20% 

30% 

23% 

29% 

n/a 

Met  with  an  agency  outside 
DIG 

5% 

29% 

0% 

16% 

57% 

n/a 

Submitted  written 
recommendations  for  policy 
changes 

38% 

9% 

10% 

13% 

43% 

n/a 

Other  follow-up 

5% 

3%, 

10% 

26% 

14% 

n/a 

n/a 

No  follow-up 

38% 

14% 

40% 

16% 

0% 

n/a 

No  answer 

0% 

0%. 

0% 

6% 

0% 

n/a 

Total 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 
N  =104 

Note:  Rows  do  not  sum  to  100%.  Columns  do  not  sum  to  100%  because  more  than  one  kind  of  follow-up  could  be  pursued 
on  each  audit. 

Source:  Abt  Associates,  1997 


Exhibit  8-17 

Survey  of  Audits:  How  Would  the  Post-ORT  Audit  Have  Been  Different  If  It  Were  Pre-ORT? 

Response 

Percent  of  Post-ORT  Audits 

Different  audit  focus 

47% 

Collaboration  in  some  form 

35% 

Audit  would  not  have  been  done  before  ORT 

21% 

Data-driven  methods  for  identifying  subject 

16% 

Resources  -  Travel 

11% 

More  active  follow-up  under  ORT 

8% 

More  unusual  methods  of  audit  development  under  ORT 

7% 

Resources  -  Other 

7% 

Resources  -  Staff 

6% 

Other  aspects  of  identification  different  under  ORT 

3% 

Management  commitment 

3% 

No  pre/post  differences 

14% 

No  answer 

6% 

Total 

n/a 

N  =103 

Note:  Respondents  were  given  open-ended  opportunity  to  respond,  so  some  mentioned  multiple  differences.  As  a  result,  rows  are 
not  mutually  exclusive  and  percentages  do  not  sum  to  100%.  Note  also  one  survey  was  excluded,  due  to  a  statement  that 
this  was  a  pre-ORT  audit  -  a  statement  that  makes  it  impossible  to  determine  the  meaning  of  answers  about  how  the  audit 
then  differed  from  a  pre-ORT  audit. 


Source:  Abt  Associates.  1997 
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•  Before  ORT.  this  audit  would  have  had  a  different  focus  [would  not  have  been  provider  specific, 
or  focused  on  one  state]  -  For  almost  one-half  of  all  audits  (47%),  respondents  suggested  that 
the  audits  would  have  had  a  different  focus  before  ORT.  Pre-ORT  audits  tended  to  be  based  on 
multi-state,  multi-provider  samples,  with  a  program  focus  rather  than  a  provider  focus,  unlike 
audits  in  the  ORT  period. 

•  Wouldn't  have  done  this  audit  before  ORT  -  In  a  similar  vein,  respondents  for  2 1  %  of  the  audits 
suggested  -  more  or  less  without  elaboration  -  that  the  audit  simply  would  not  have  been  done 
before  ORT.  This  judgment  could  be  another  form  of  the  "different  focus"  claim,  but  it  could 
also  be  referring  to  the  subject-type  focus  of  ORT  (DME,  home  health,  and  nursing  home)  or 
even  the  resources  of  ORT. 

•  Before  ORT,  we  would  not  have  had  collaboration  -  As  noted  earlier  (see  discussion  of 
Questions  6  and  7),  ORT  audits  involved  collaboration  in  initiation  over  80%  of  the  time  and 
in  development  over  80%  of  the  time.  From  a  different  angle  -  when  asked  in  this  question  how 
the  ORT  audit  before  them  would  have  been  different  before  ORT  -  respondents  claim  that  the 
collaboration  that  took  place  in  fact  would  not  have  taken  place  in  35%  of  the  cases.  As  noted 
earlier  (see  discussion  of  Question  7),  this  figure  constitutes  a  rough  lower  bound  on  the 
estimated  incremental  effects  of  ORT  on  collaboration  in  ORT-type  audits. 

•  Audit  wouldn't  have  been  data-driven  before  ORT:  In  a  variety  of  different  ways,  respondents 
suggested  that  audit  methods  would  have  been  different  in  that  they  would  not  have  been  so 
data-driven.  This  suggestion  was  made  for  16%  of  all  audits. 

•  Travel  resources  wouldn't  have  been  available  before  ORT  -  Similarly,  respondents  suggested 
that  travel  resources  would  not  have  been  available  in  11  %  of  the  audits. 

•  The  audit  wouldn't  have  been  different  before  ORT  -  One  in  seven  ORT  audits  were  thought 
to  have  been  little  different  from  a  typical  pre-ORT  audit. 

These  observations  provide  a  revealing  summary  appraisal  of  ORT  and  OAS,  an  appraisal  that  is 
consistent  with  the  answers  to  other  questions  we  have  addressed.  In  these  views,  ORT  represents  a 
change  in: 

•  approach,  toward  more  provider-specific  audits, 

•  process,  toward  more  collaboration. 

•  techniques,  toward  more  data-driven  methods,  and 

•  resources,  toward  more  needed  funding,  particularly  in  travel. 

This  overview  of  how  the  audits  would  have  been  different  is  a  reasonable  overview  of  what  ORT  tried 
to  accomplish. 
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It  is  interesting  to  note  the  contrast  to  the  results  for  the  survey  of  cases  (Appendix  7).  For  investigators, 
the  big  difference  ORT  made  was  in  collaboration  and  resources.  The  implication  is  that  OI  investigators 
saw  ORT  as  a  ramping  up  of  slightly  changed  enforcement  methods,  while  OAS  auditors  saw  ORT  as 
more  of  an  innovation  in  technique  and  approach.  That  result  comports  more  or  less  exactly  with  the 
findings  of  our  process  analyses  (Appendix  1 ). 

8.5     Auditor-specific  Survey  Results 

A  final  way  in  which  we  sought  to  use  auditors'  opinions  to  understand  ORT  was  to  piggy-back  auditor- 
specific  questions  on  our  audit-specific  survey.  Specifically,  in  order  to  gain  a  better  understanding  of 
auditors'  overall  opinions  of  ORT,  we  asked  the  auditors  to  look  beyond  the  audit  at  hand  and  to  elaborate 
on  their  views  of  ORT.  We  had  interviewed  OAS  auditors  during  site  visits  but  wanted  to  confirm, 
through  a  confidential  questionnaire,  their  opinions. 

8.5.1  Nature  of  the  Questions 

As  noted  in  Exhibit  8-3  above,  we  posed  four  questions  to  auditors:  what  has  worked  well  in  ORT,  what 
has  not,  and  what  parts  of  ORT  should  and  should  not  be  adopted.  Answers  were  un-prompted  (there  were 
no  check  boxes).  Ample  space  was  provided  so  that  auditors  could  respond  freely.  Their  open-ended 
responses  were  then  coded  into  a  pre-developed  coding  scheme.  These  subjective  appraisals  of  ORT  give 
us  some  insight  into  how  field  personnel  view  ORT. 

8.5.2  Sample 

While  we  received  106  usable  audit  surveys  (2  pre-ORT  and  104  post-ORT),  opinion  questions  are 
auditor-specific,  not  audit-specific.  Thus,  survey  instructions  directed  auditors  to  answer  the  overall 
opinion  questions  only  once,  so  as  not  to  duplicate  answers.  Where  individual  auditors  inadvertently 
completed  this  section  more  than  once  (i.e.,  when  the  questions  were  answered  on  more  than  one  survey 
by  what  clearly  was  one  auditor),  responses  were  entered  only  once.  However,  while  our  instructions  were 
specific  and  our  review  of  responses  careful,  we  cannot  be  sure  that  we  avoided  all  duplicates.  For  that 
reason,  the  results  of  the  auditor-specific  questions  should  be  viewed  as  approximations  only. 

The  N  for  each  question  is  based  on  the  number  of  different  auditors  that  responded  to  that  question,  not 
the  number  of  surveys  received.  Exhibit  8-18  summarizes  the  N  by  question. 


Exhibit  8-18 

Survey  of  Auditors:  Number  Who  Responded  to  Opinion  Questions 

Question  N 

1 7.  What  in  ORT  has  worked  well?  21 

18.  What  in  ORT  has  not  worked  well?  16 
19a.  What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt?  18 
19b.  What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopt?  10 
Source:  Abt  Associates,  1997   
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8.5.3      Auditors'  Responses 


What  in  ORT  has  worked  well?  (Question  17) 

As  shown  on  Exhibit  8-19.  81<7f  of  respondents  to  this  question  report  that  the  success  of  ORT  can  be 
attributed  to  partnerships,  both  among  ORT  agencies  and  with  non-ORT  agencies.  According  to  the 
auditors,  collaboration  increased  efficiency  in  audit  and  enforcement  processes.  A  variety  of  different 
forms  of  collaboration  were  mentioned,  including  cross-region  partnerships  and  increased  communication 
and  meetings  with  01,  OIG,  HCFA,  the  Medicare  contractors.  Sharing  information  with  other  OAS 
regions  was  said  to  reduce  duplicate  efforts  and  streamline  targeting  initiatives.  HCFA's  provision  of 
medical  assistance  and  reviews  of  Medicare  cost  reports  were  mentioned  explicitly  as  appreciated 
collaboration  efforts. 

According  to  the  auditors,  the  commitment  of  added  resources  played  the  second  most  important  role  in 
making  ORT  work  well.  To  differentiate  among  different  types  of  resources,  we  coded  responses 
mentioning  resources  into  "travel,"  "staff,"  or  "other"  resources,  depending  upon  how  the  comment  was 
worded.  Nearly  24%  of  the  auditors  pointed  to  travel  funds  as  one  of  the  reasons  ORT  worked  well. 
Without  the  travel-designated  monies,  auditors  claimed,  audits  might  have  been  delayed  or  might  not  have 
been  performed  at  all.  Travel  funds  were  also  said  to  have  increased  the  likelihood  of  cross-region 
collaboration.  When  elaborating  on  other  helpful  resources,  auditors  mentioned  computers  and  equipment, 
along  with  office  space  and  funds  in  general.  Added  staff  were  infrequently  mentioned,  but  then  OAS 
received  only  a  small  infusion  of  staff  under  ORT.  As  seen  in  Exhibit  8-19,  top  management  commitment 
and  ORT's  change  in  focus  to  provider-specific  targeting  were  also  mentioned  by  auditors  as  positive 
attributes  of  ORT. 

What  in  ORT  has  not  worked  well?  (Question  18) 

Exhibit  8-20  shows  what  auditors  felt  had  not  worked  well  in  ORT.  While  most  auditors  believe  the 
partnership  model  was  a  major  accomplishment  of  ORT,  there  is  apparently  room  for  improvement.  Of 
the  16  auditors  responding  to  this  question,  31%  suggested  that  more  steps  were  needed  to  improve 
cooperation  and  collaboration  among  the  various  partners.  The  collaboration  problems  mentioned  were 
diverse:  one  auditor  thought  cooperation  with  01  needed  improvement;  a  second  auditor  said  that 
collaboration  with  01  as  the  lead  organization  had  not  worked  well;  a  third  auditor  noted  that  OAS/HCFA 
collaboration  had  actually  delayed  some  audits;  another  auditor  suggested  that  coordination  with  HCFA 
Regional  Offices  tended  to  delay  audit  results;  and  a  final  auditor  said  the  Department  of  Justice  was  not 
cooperative  in  pursuing  referrals.  Thus,  while  one-third  of  auditors  see  room  for  improvement  in 
collaboration,  no  single  area  of  collaboration  was  identified  as  the  problem. 

Some  of  these  comments  could  perhaps  be  explained  by  auditors'  references  to  the  intrinsic  or  "structural" 
problems  associated  with  enforcement  processes  (257c  of  the  auditors  noted  such  problems).  ORT  cannot 
repeal  the  constitutional  and  other  protections  suspect  providers  enjoy,  and  ORT  did  not  change  the  (at 
times  frustratingly  complex)  division  of  labor  among  different  organizations  in  the  enforcement  process. 
But  ORT  efforts  can  surely  be  complicated  by  such  limits,  and  one-quarter  of  all  auditors  noted  that 
frustration. 

A  number  of  auditors  noted  problems  with  ORT  administrative  overhead  and  procedures.  Specifically, 
some  auditors  objected  to  such  things  as  the  frequency  of  meetings,  the  level  of  monitoring  and  reporting 
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Exhibit  8-19.  Survey  of  Auditors: 

What  in  ORT  Has  Worked  Well? 
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Exhibit  8-20.  Survey  of  Auditors: 

What  in  ORT  Has  Not  Worked  Well? 


100% 
90% 
80% 
70% 


60% 

13 


ORT  Partner  Collaboration  Don't  Know  ORT  structural  problems  Resources  still  too  limited 

Intrinsic  structure  problems  Administrative  burden  Restrictions  on  ORT  partners 

Response  Category 

Note:  N  =  16  I 
Source:  Abt  Associates,  1997 
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requirements,  the  difficulties  of  separating  ORT  and  non-ORT  budgets,  and  the  restriction  of  ORT  funds 
to  ORT-type  jobs. 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  to  adopt?  (Ouestion  19a) 

Not  surprisingly,  auditors'  recommendations  to  other  regions  were  similar  to  auditors'  judgments  of  what 
worked  well  in  ORT.  Collaboration  was  viewed  as  the  chief  benefit  of  ORT  that  non-ORT  regions  should 
adopt.  As  depicted  in  Exhibit  8-21,  789^  of  the  18  auditors  who  responded  to  this  question  believed 
partnerships  were  the  key  to  the  success  of  ORT-style  endeavors.  Without  the  partnership  model,  it  was 
said,  agencies  and  contractors  would  not  be  sharing  ideas,  expertise,  resources,  and  information  .  Joint 
reviews,  teaming  with  state  licensing  and  certification  boards,  and  cross-region  coordination  were 
mentioned  by  auditors  as  being  important  parts  of  ORT  for  new  regions  to  replicate. 

Other  aspects  of  ORT  mentioned  by  auditors  included  travel  funds,  the  commitment  of  high-level 
management,  change  in  focus  to  provider-specific  audits,  and  the  adoption  of  more  data-driven  methods. 
These  responses  are  consistent  with  our  audit-based  results. 

What  single  part  of  ORT  would  you  urge  non-ORT  regions  not  to  adopt?  (Question  19b) 

We  received  fewer  responses  to  this  question  (10)  than  to  the  other  auditor-specific  questions  we  asked. 
This  reduction  in  responses  could  be  a  product  of  many  different  factors: 

•  The  "last  question"  effect  -  When  a  respondent  has  completed  numerous  surveys,  the  last 
question  tends  to  receive  less  attention. 

•  Relatively  few  negative  opinions  of  ORT  -  Many  auditors  may  believe  that  there  is  no  part  of 
ORT  that  was  so  ill  advised  that  it  should  not  be  tried  in  other  regions.  As  a  result,  these 
auditors  may  simply  have  deliberately  skipped  the  question. 

The  responses  presented  in  Exhibit  8-22  tend  to  confirm  the  latter  possibility.  While  10  auditors  answered 
the  question,  two  auditors  believed  that  there  was  no  part  of  ORT  that  should  be  overiooked  in  expansion 
to  another  region.  Similarly,  two  auditors  claimed  that  they  knew  of  no  aspects  of  ORT  that  should  not 
be  adopted  by  non-ORT  regions.  (These  were  not  the  same  auditors  who  answered  that  no  part  of  ORT 
should  be  overlooked.)  These  results  suggest  that  four  of  the  ten  auditors  responding  believed  that  most 
parts  of  ORT  worked  well  enough  to  warrant  replication  in  non-ORT  regions. 

Of  the  six  auditors  who  had  reservations  about  replicating  parts  of  ORT,  three  suggested  that  the  ORT 
partnership/collaboration  approach  should  not  be  adopted.  For  example,  one  auditor  said  that 
"partnerships  did  not  work"  and  instead  managers  should  strive  for  "cooperative  relationships."  Another 
auditor  said  that  there  was  too  much  "dependence  on  01."  but  did  not  detail  how  this  was  a  drawback. 

One  auditor  thought  that  there  was  excessive  administrative  time  involved  with  obtaining  and  verifying 
data.'^  Separation  of  resources  among  ORT  and  non-ORT  monies  were  also  mentioned  as  being  annoying 
processes  of  ORT.  Of  course,  this  latter  problem  is  common  in  demonstrations  -  indeed,  segregation  of 


14     Perhaps  the  problem  with  data  collection/verification  was  a  reflection  of  resource  limitations  -  i.e..  if  staff  or  other  resources  were 
cntically  hmited  at  OAS  or  at  the  agency  providing  the  data,  data  collection  and  verification  could  be  frustrating 


Abt  Associates  Inc. 


Appendix  8:  Findings  from  the  Survey  of  Audits 


8-28 


Exhibit  8-21 

Survey  of  Auditors:  What  single  part  of  ORT  should  be  adopted  by  non-ORT  regions? 


%  of  Auditors  Who 


Response  Category 

Responded 

Collaboration  in  some  form 

14 

78% 

Travel  funds 

2 

11% 

Management  commitment 

2 

11% 

Different  audit  focus 

2 

11% 

Don't  know 

2 

11% 

Other 

2 

11% 

Data-driven  methods  for  identifying  subject 

1 

6% 

More  unusual  methods  of  audit  development  under 
ORT 

1 

6% 

Other  Resources 

1 

6% 

Total  Auditors  who  responded:  18 

1       Though  the  question  asks  the  respondent  what  single  part  of  ORT  should  be  adopted  by  non-ORT  regions,  some  auditors 
listed  more  than  one  aspect.  Therefore,  the  number  of  auditors  responding  (18)  does  not  equal  the  number  of  responses 
(27). 

Source:  Abt  Associates,  1997 

Exhibit  8-22 

Survey  of  Auditors:  What  single  part  of  ORT  should  nof  be  adopted  by  non-ORT  regions? 

Response  Category 

N' 

%  of  Auditors  Who 
Responded 

ORT  partner  collaboration 

3 

30%. 

Nothing 

2 

20% 

Unknown 

2 

20% 

Resources  still  too  limited 

1 

10% 

Excessive  administrative  duties 

1 

10% 

ORT  structure  restrictive 

1 

10% 

Other 

1 

10% 

Total  Auditors  who  responded:  10 


'  Though  the  question  asks  the  respondent  what  single  part  of  ORT  should  not  be  adopted  by  non-ORT  regions,  some 
auditors  listed  more  than  one  aspect.  Therefore,  the  number  of  auditors  responding  (10)  does  not  equal  the  number  of 
responses  (11). 

Source;  Abt  Associates,  1997   
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funds  is  commonly  a  price  of  doing  a  demonstration  -  but  there  probably  is  not  much  long-run  significance 
to  the  difficulty. 

Summary  of  the  Auditor-Specific  Comments 

Clearly,  when  they  speak  about  ORT  overall,  auditors  see  collaboration  and  resources  as  the  greatest 
benefits  of  the  initiative.  In  reviewing  auditors"  responses,  collaboration  has  lead  to  some  collaboration 
among  01,  OAS,  HCFA,  OGC,  FIs  and  state  agencies,  and  to  such  practices  as  joint  reviews  with 
DMERCs,  OAS  assistance  with  01  cases,  task  force  reviews  with  DoJ  and  the  FBI,  PRO  support  for 
medical  review,  and  cross-region  teamwork.  Meanwhile,  increased  funding  under  ORT  has  allowed  for 
increased  travel  funds,  data  assistance,  and  other  needed  functions. 

Turning  the  question  around,  of  the  21  auditors  who  expressed  their  views  on  ORT,  only  about  half  had 
complaints  about  ORT.  Overall,  these  complaints  of  ORT  were  fairly  minimal  and  do  not  offer  much 
instruction  on  how  to  improve  ORT.  Auditors  believe  that  there  is  room  for  improvement  in  the  area  of 
collaboration  and  that  resources  are  still  limited,  but  there  in  fact  may  be  a  difference  of  opinion  among 
auditors  on  how  this  should  be  accomplished. 


Abt  Associates  Inc. 


Appendix  8:  Findings  from  the  Survey  of  Audits 


8-30 


Attachments 


DEPARTMENT  OF  HEALTH  &  HUMAN  SERVICES 


Office  of  Insoector  General 


Wasnmgton,  D.C.  20201 


May  1,  1997 
[Auditor  Address] 

RE:     OPEM  TION  RESTORE  TR  UST:  Abt  Associates  '  Survey  of  OAS  A  udits 
Dear  [Auditor]: 

Abt  Associates  is  under  contract  with  the  Health  Care  Financing  Administration  (HCFA)  to 
evaluate  Operation  Restore  Trust  (ORT).  As  part  of  that  evaluation.  Abt  would  like  to  explore  the 
effects  of  ORT  on  a  sample  of  pre-  and  post-ORT  audits,  in  order  to  estimate  how  ORT  has  affected 
OAS  audit  activities.  This  systematic  data  collection  will  provide  qualitative  information  about 
specific  audits  —  information  not  supplied  by  our  Audit  Information  Management  System  (AIMS) 
or  discussed  during  interviews  you  may  have  had  with  Abt  staff  over  the  past  year.  This  information 
will  permit  Abt  to  provide  a  richer,  better  documented  picture  of  all  you  have  done  under  Operation 
Restore  Trust. 

The  enclosed  instrument — Survey  of  a  Sample  of  OAS  Audits:  Interview  Questions —  poses 
key  questions  about  the  details  of  a  specific  audit  in  which  you  played  a  key  role.  Instructions  for 
completing  the  survey  are  contained  in  the  instrument  and  accompanying  letter  from  Abt. 

We  encourage  you  to  complete  the  questionnaire  within  the  next  week.  The  survey  is 
confidential:  it  should  be  returned  directly  to  Abt  Associates.  We  will  not  be  reviewing  the 
individual  survey  responses. 

If  you  have  any  questions,  please  contact  John  Hapchuk.  Director  of  Programs  and 
Operations  Audits,  at  (202)  205-91 12.  Questions  about  the  survey  instrument  should  be  directed 
to  Ms.  Selene  Angier,  Survey  Director  at  Abt  Associates,  at  (800)  265-9591. 

We  appreciate  your  attention  to  this  very  important  matter.  With  your  cooperation,  we  will 
provide  Abt  with  the  information  it  needs  to  document  our  important  accomplishments  under  ORT. 


Sincerely, 


Thomas  D.  Roslew  icz 
Deputy  Inspector  General 


cc:      John  Hapchuk.  OAS 


Abt 


Abl  Assoeiate^i  Inc. 


May  1,  1997 
[Auditor  Address] 
Dear  [Auditor]: 

In  April  1995,  the  Departmentof  Health  and  Human  Services,  along  with  several  other  federal  and 
state  agencies,  launched  Operation  Restore  Trust  (ORT).  Abt  Associates  is  under  contract  to  the 
Health  Care  Financing  Administration  (HCFA)  to  evaluate  the  ORT  demonstration.  In  order  to 
explore  how  ORT  has  affected  the  identification,  developmentand  outcomes  of  OAS  audits,  we  are 
surveying  auditors  about  pre-ORT  audits  (i.e.,  audits  closed  before  April  1995)  and  ORT  period 
audits  (i.e.,  audits  closed  after  April  1995). 

To  date,  we  have  interviewed  OAS  auditors  about  ORT,  but  have  not  collected  systematic  qualitati\e 
mformation  about  a  sample  of  specific  audits.  The  enclosed  survey  is  designed  for  that  purpose.  We 
would  like  you  to  complete  this  survey  in  order  to  help  us  answer  key  questions  about  some  of  the 
major  accomplishmentsof  ORT  such  as:  Are  audits  differently  targeted  under  ORT?  Are  particular 
resources  especially  helpful?  Has  ORT  changed  the  timing  and  level  of  collaboration  among  kev 
participants?  With  your  help,  we  will  be  able  to  determine  whether  the  answers  to  these  questions 
differ  among  pre-  and  ORT  period  audits. 

Enclosed  please  find  the  Survey  of  a  Sample  of  OAS  Audits:  Inleniew  Questions.  We  ask  that  you 
complete  the  questionnaire  within  the  next  week.  Please  return  it  directly  to  us  using  the  enclosed 
pre-addressed, postage-paid  envelope.If  you  have  any  questions,  please  feel  free  to  contact  Selene 
Angier,  Survey  Coordinator,at  (800)  265-9591  or  viainternet  E-mail:  selene_angier@abtassoc.com 

We  appreciate  your  time  and  effort  in  filling  out  this  very  important  survey. 

Sincerely. 


Robert  F.  Couiam,  Ph.D. 
Project  Director 
Abt  Associates 


cc:  John  Hapchuk,  OAS 


55  Wheeler  Street  ■  Cambridge.  .Massachusetts  USA  ■  02138-1168  ■  617  492-7100  rc/fp^oHc  ■  6\~  ■\'41-i2\^  facsimile 


Notes  to  Survey  Hypothetical 
Q  16.  -  Survey  of  Audits 

In  order  to  prevent  confusion,  we  designed  two  instruments —  a  pre-ORT  version  and  a  post-ORT 
version  —  for  the  Survey  of  Audits. 

The  following  version  of  the  Survey  of  Audits  is  designed  for  pre-ORT  audits  (i.e.,  audits  that  closed 
before  ORT  began  on  April  1,  1995).  In  the  post-ORT  version,  the  Hypothetical  question  is  worded  as 
follows: 

This  audit  was  open  at  least  some  time  after  ORT  began  on  April  1,  1995.  How  would  this  audit 
have  been  different  if  it  had  been  completed  before  ORT? 


SURVEY  OF  A  SAMPLE  OF  OAS  AUDITS: 
 INTERVIEW  QUESTIONS 


Abt  Associates  is  under  contract  with  the  Health  Care  Financing  Administration  (HCFA)  to  carry  out  the 
Evaluation  of  Operation  Restore  Trust.  As  part  of  our  evaluation,  we  are  surveying  OAS  auditors  in  order 
to  better  understand  the  initiation,  development,  and  outcomes  of  pre-  and  post-ORT  audits.  Please  be 
aware  that  any  information  obtained  through  this  survey  is  confidential.  Your  identity  will  not  be  made 
known  to  anyone  outside  of  Abt  Associates,  in  any  published  reports  or  in  any  other  form.  Your  completed 
questionnaires  are  to  be  sent  directly  to  Abt  Associates.  In  addition  to  answering  the  questions  posed  below, 
please  verify  and,  if  necessary,  correct  the  following  information: 

Audit  ID#: 
Audit  Description: 
State/Region: 
Subject  Type: 
Year  Audit  Opened: 
Period: 

Status  as  of  8/96: 
Audit  Manager: 


Your  role  in  the  audit: 


Audit  Purpose: 

1 .  What  was  the  primary  purpose  of  the  audit?  [Check  one  only.] 

□  Identify  or  target  aberrant  providers  for  audit  and/or  OI  referral 

□  Identify  questionable  charges  from  specific  provider(s)  for  recoipment/other  foUowup  actions 

□  Assist  an  active  investigation 

□  Appraise  policies,  practices  and  vulnerabilities  of  the  Medicare  program 

□  Other —  Explain:   


Audit  target  or  subject: 

2.  What  prompted  the  audit?  How  were  the  subject(s)  of  the  audit  identified?  [Check  one  only.] 

□  Self-initiated  by  OAS —  Explain:  

□  Referral  from  OAS  in  another  region 

□  Referral  from  HCFA 

□  Referral  from  HCFA  contractor 

n   Referral  from  MFCU  or  other  state  agency 

□  Request  from  01 

□  1  don't  know 

□  Other —  Explain:  

3.  Were  any  special  analyses,  data  runs,  or  other  efforts  undertaken  to  identify  the  subject(s)  of  the 
audit?  If  so,  what  were  those  special  efforts  and  who  performed  them? 

□  No         □  Yes-  Specify:  
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4.   By  comparison  to  audits  before  ORT  (i.e..  before  April  1,  1995),  was  the  initiation  of  this  audit 
unusual  or  atypical  in  any  of  the  following  ways?/C/iec/t  all  thai  apply./ 

□  No        Yes-  Explain:    □   Number  of  targeted  providers?   

□  Location  of  targeted  providers?  

□  Complexity  of  suspected  scheme,  abuse  or  infraction?  


□  Complexity  of  audit? 

□  Other  

□  Not  applicable,  this  was  a  pre-ORT  audit 

5.  Was  there  anything  else  unusual  about  this  audit? 

□  No  □  Yes-  Specify:  


Collaboration: 


With  respect  to  the  identification  of  the  audit  subject(s),  did  you  have  any  notable  collaboration 
with:  [Check  all  that  apply. J 

Federal  Government 

□  01?  □   OAS  in  another  region? 

□  OEI?  □   U.S.  Attorney's  office? 

□  OEC?  □   Medicare  contractors? 

□  OLC?  □   Peer  review  organization? 

□  HCFA? 

□  AoA? 

□  FBI? 

□  Other  federal  contractor  or  agency?  Specify:  

State  Government 

□  Long-term  care  ombudsmen? 

□  Medicaid  agency? 

□  MFCUs? 

□  State  survey  or  certification  staff? 

□  Prosecutors? 

□  Other  state  contractor  or  agency?  Specify:  \  

□  Qui  tarn? 

□  Hot  Line? 

□  Other?  Specify:   

□  No  notable  collaboration  with  respect  to  identifying  the  subject{s) 

Briefly  describe  the  nature  of  the  collaboration,  if  any: 
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7.  With  respect  to  the  development  of  the  audit  itself,  did  you  have  any  notable  collaboration  with: 

[Check  all  that  apply.] 


Federal 


Government 

on 


□ 
□ 
□ 
□ 


OEI? 

OEC? 

OLC? 


□ 
□ 
□ 
□ 


OAS  in  another  region? 
U.S.  Attorney's  office? 
Medicare  contractors? 


Peer  review  organization? 


□  HCFA? 

□  AoA? 

□  FBI? 

□  Other  federal  contractor  or  agency?  Specify;  

State  Government 

□  Long-term  care  ombudsmen? 

□  Medicaid  agency? 

□  MFCUs? 

□  State  survey  or  certification  staff? 

□  Prosecutors? 

□  Other  state  contractor  or  agency?  Specify':   

□  Other?  Specify:  

□  No  notable  collaboration  with  respect  to  audit  development 

Briefly  describe  the  nature  of  the  collaboration,  if  any: 


8.  Relative  to  the  typical  OAS  audit,  did  audit  development  require  an  unusual  amount  of: 
[Check  all  that  apply.] 

□  Staff  time? 

□  Travel? 

□  Computer  resources? 

□  Medical  review? 

□  Communication  costs? 

□  Legal  resources? 

□  Other?  

□  None  of  the  above 

9.  Was  audit  development  limited  by  the  shortage  of  any  particular  type  of  resource? 

□  No        □  Yes-  Specify:  


10.  Approximately  how  much  overnight  travel  did  this  audit  require? 

□  None  □   Some  (e.g.,  less  than  20  days)        □   A  lot  (e.g.,  20  days  or  more) 


Resources: 
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Methods: 


11.  Relative  to  the  typical  OAS  audit  conducted  before  ORT  (i.e..  before  April  1.  1995).  was  there 
an>lhing  unusual  about  how  this  audit  was  conducted?  [Check  all  that  apply.] 

□  No,  nothing  unusual       Yes-Explain:    □  Notably  sophisticated  technique?  

□  Unusual  audit  approach?  

□  Other-  Specify:  

□  Not  applicable,  this  was  a  pre-ORT  audit 
Results  (Insofar  as  you  know  them): 

12a. For  completed  audxXs:  Has  a  final  audit  report  been  issued? 

□  Yes  □  No 

12b.  For  audits  currently  pending:    Do  you  expect  a  final  report  to  be  issued? 

□  Yes  □  No 

13.  With  respect  to  this  audit,  have  any  of  the  following  outcomes  occurred?  [Check  all  that  apply.] 

□  Referral  to  01 

□  Referral  to  FBI 

□  Referral  to  OEC 

□  Referral  to  OLC 

□  Formal  criminal  charges 

□  Formal  civil  charges 

□  Exclusion 

□  Recoupment 

□  Identification  of  other  providers  to  be  audited 

□  Other —  Specify:  

□  None  of  these  outcomes  has  occurred 

□  Don't  know 

14.  Did  you  take  any  of  the  followup  actions  listed  below  as  a  result  of  the  audit?  [Check  all  that  apply. [ 

□  Commenced  audits  of  similar  provider(s) 

□  Submitted  written  recommendations  for  policy  changes  (e.g.,  changes  in  federal  statutes  or 
regulations)  Specify   


□  Met  with  an  agency  outside  OIG.  Which  agenc(ies)?  

□  Met  with  a  Medicare  contractor  or  made  written  recommendation  to  change  some  aspect  of 
contractor  operations 

□  Other  


□  No  followup  actions  taken 

15.  Did  the  audit  result  in  the  identification  of  potential  questioned  costs?  If  so,  approximately  how 
much  in  questioned  costs  were  identified? 

□  No  □   Yes  (approximate  amount):  $   

□  Too  soon  to  say 
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Hypothetical: 


16.  For  the  purposes  of  this  survey,  this  audit  is  considered  a  pre-ORT  audit.  How  would  this  audit 
have  been  different  under  ORT? 


Overall  Opinions  on  ORT: 

Senior  OIG  managers  are  particularly  interested  in  your  views  on  how  ORT  has  generally  affected  your 
work.  For  that  purpose,  the  questions  below  look  beyond  this  one  audit  and  ask  you  to  compare  the 
ORT  and  pre-ORT  periods  generally.  (If  you  are  completing  surveys  for  more  than  one  audit  and  have 
already  answered  these  questions  on  another  survey  regarding  a  different  audit,  you  do  not  have  to 
repeat  your  answers  here.  But  please  be  sure  to  answer  the  questions  at  least  once.) 

17.  What  in  ORT  has  worked  well? 


18.  What  in  ORT  has  not  worked  well? 


19.  If  you  were  addressing  a  group  of  your  colleagues  from  non-ORT xtgions,  what  single  part  of  ORT 
would  you  urge  them: 

a.   To  adopt? 


b.    Not  to  adopt? 
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